[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:56 dist-test-slave-g1xh krb5kdc[2744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:56 dist-test-slave-g1xh krb5kdc[2744](info): set up 2 sockets
Sep 25 02:49:56 dist-test-slave-g1xh krb5kdc[2744](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 02:49:57.540370 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.008s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:49:57 dist-test-slave-g1xh krb5kdc[2744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:49:57.740633 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:44921
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:58.167246 2757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:58.167734 2757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:58.167891 2757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:58.175871 2757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:58.176016 2757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:58.176106 2757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:58.176198 2757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:58.176280 2757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:58.187520 2757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:44921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:58.189409 2757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:58.191305 2757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:58.208321 2765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:58.208400 2763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:58.208619 2762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:58.209620 2757 server_base.cc:1034] running on GCE node
I20260925 02:49:58.210631 2757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:58.212636 2757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:58.213871 2757 hybrid_clock.cc:648] HybridClock initialized: now 1790304598213783 us; error 77 us; skew 500 ppm
I20260925 02:49:58.214354 2757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:58 dist-test-slave-g1xh krb5kdc[2744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:58.217833 2757 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:49:58.219811 2757 webserver.cc:460] Webserver started at http://127.2.172.126:36913/ using document root <none> and password file <none>
I20260925 02:49:58.220719 2757 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:58.220832 2757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:58.221175 2757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:58.224828 2757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "63ecbc816246466fb5904aee4c61a269"
format_stamp: "Formatted at 2026-09-25 02:49:58 on dist-test-slave-g1xh"
server_key: "ac06c05c61a618658ed1bdffcd64babc"
server_key_iv: "f051fb3d1ba60904ddd5cfdcf88bfc32"
server_key_version: "encryptionkey@0"
I20260925 02:49:58.225600 2757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "63ecbc816246466fb5904aee4c61a269"
format_stamp: "Formatted at 2026-09-25 02:49:58 on dist-test-slave-g1xh"
server_key: "ac06c05c61a618658ed1bdffcd64babc"
server_key_iv: "f051fb3d1ba60904ddd5cfdcf88bfc32"
server_key_version: "encryptionkey@0"
I20260925 02:49:58.232286 2757 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 02:49:58.236968 2772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:58.238672 2757 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:49:58.238910 2757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "63ecbc816246466fb5904aee4c61a269"
format_stamp: "Formatted at 2026-09-25 02:49:58 on dist-test-slave-g1xh"
server_key: "ac06c05c61a618658ed1bdffcd64babc"
server_key_iv: "f051fb3d1ba60904ddd5cfdcf88bfc32"
server_key_version: "encryptionkey@0"
I20260925 02:49:58.239158 2757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:58.268132 2757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:58.273682 2757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:58.274029 2757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:58.305687 2757 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:44921
I20260925 02:49:58.305727 2825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:44921 every 8 connection(s)
I20260925 02:49:58.307468 2757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:58.312583 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 2757
I20260925 02:49:58.312912 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:49:58.313939 2826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:58.315099 2737 external_mini_cluster.cc:1428] Setting key 862cea764b8c324fa4fb97d5e74e9096
I20260925 02:49:58.332592 2826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269: Bootstrap starting.
I20260925 02:49:58.338912 2826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:58.340749 2826 log.cc:824] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:58.345615 2826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269: No bootstrap required, opened a new log
Sep 25 02:49:58 dist-test-slave-g1xh krb5kdc[2744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:49:58.358470 2826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ecbc816246466fb5904aee4c61a269" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44921 } }
I20260925 02:49:58.358948 2826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:58.359112 2826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ecbc816246466fb5904aee4c61a269, State: Initialized, Role: FOLLOWER
I20260925 02:49:58.360210 2826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ecbc816246466fb5904aee4c61a269" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44921 } }
I20260925 02:49:58.360586 2826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:58.360888 2826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:58.361158 2826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:58.365311 2826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ecbc816246466fb5904aee4c61a269" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44921 } }
I20260925 02:49:58.366284 2826 leader_election.cc:302] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63ecbc816246466fb5904aee4c61a269; no voters:
I20260925 02:49:58.367168 2826 leader_election.cc:288] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:58.367429 2831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:58.370987 2826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:58.371855 2831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [term 1 LEADER]: Becoming Leader. State: Replica: 63ecbc816246466fb5904aee4c61a269, State: Running, Role: LEADER
I20260925 02:49:58.372565 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:58.320129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.2.172.126:44921)
0925 02:49:58.321718 (+ 1589us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:58.321735 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:58.325559 (+ 3824us) server_negotiation.cc:407] Connection header received
0925 02:49:58.340471 (+ 14912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:58.340499 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:58.341024 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:58.341534 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:58.346223 (+ 4689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:58.347686 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:58.350460 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:58.351228 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:58.351431 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:58.359887 (+ 8456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:58.359901 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:58.359926 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:58.359988 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:58.364822 (+ 4834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:58.365936 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:58.365947 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:58.365957 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:58.366344 (+ 387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:58.367024 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:58.367030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:58.367033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:58.368044 (+ 1011us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:58.368520 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:58.370090 (+ 1570us) server_negotiation.cc:299] Negotiation successful
0925 02:49:58.370555 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1197,"thread_start_us":588,"threads_started":1}
I20260925 02:49:58.372684 2831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ecbc816246466fb5904aee4c61a269" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44921 } }
I20260925 02:49:58.386361 2832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63ecbc816246466fb5904aee4c61a269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ecbc816246466fb5904aee4c61a269" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44921 } } }
I20260925 02:49:58.387149 2832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:58.386564 2833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63ecbc816246466fb5904aee4c61a269. Latest consensus state: current_term: 1 leader_uuid: "63ecbc816246466fb5904aee4c61a269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ecbc816246466fb5904aee4c61a269" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44921 } } }
I20260925 02:49:58.387645 2833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:58.392652 2844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:58.400424 2844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:58.412642 2844 catalog_manager.cc:1347] Generated new cluster ID: 39d620d9deb94c82a8f2d8224fd31250
I20260925 02:49:58.412777 2844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:58.438266 2844 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:58.439836 2844 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:58.447777 2844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63ecbc816246466fb5904aee4c61a269: Generated new TSK 0
I20260925 02:49:58.449231 2844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:49:58.504386 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:44921
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:58.864012 2852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:58.864324 2852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:58.864491 2852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:58.872228 2852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:58.872390 2852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:58.872632 2852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:49:58.883033 2852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:44921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:58.884531 2852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:58.885993 2852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:58.900849 2857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:58.900876 2858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:58.900924 2860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:58.901978 2852 server_base.cc:1034] running on GCE node
I20260925 02:49:58.902685 2852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:58.903935 2852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:58.905187 2852 hybrid_clock.cc:648] HybridClock initialized: now 1790304598905138 us; error 45 us; skew 500 ppm
I20260925 02:49:58.905545 2852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:58 dist-test-slave-g1xh krb5kdc[2744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:58.909081 2852 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:49:58.910775 2852 webserver.cc:460] Webserver started at http://127.2.172.65:42843/ using document root <none> and password file <none>
I20260925 02:49:58.911533 2852 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:58.911758 2852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:58.912101 2852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:58.915323 2852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "2c462178941b4587a3abb6db6130f4f5"
format_stamp: "Formatted at 2026-09-25 02:49:58 on dist-test-slave-g1xh"
server_key: "131b0368121612e1d3a0eed9b24d9298"
server_key_iv: "f0368fb12e0d1fef6b86a8141edf04bd"
server_key_version: "encryptionkey@0"
I20260925 02:49:58.916229 2852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "2c462178941b4587a3abb6db6130f4f5"
format_stamp: "Formatted at 2026-09-25 02:49:58 on dist-test-slave-g1xh"
server_key: "131b0368121612e1d3a0eed9b24d9298"
server_key_iv: "f0368fb12e0d1fef6b86a8141edf04bd"
server_key_version: "encryptionkey@0"
I20260925 02:49:58.921898 2852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:49:58.926048 2867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:58.927796 2852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:49:58.927994 2852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "2c462178941b4587a3abb6db6130f4f5"
format_stamp: "Formatted at 2026-09-25 02:49:58 on dist-test-slave-g1xh"
server_key: "131b0368121612e1d3a0eed9b24d9298"
server_key_iv: "f0368fb12e0d1fef6b86a8141edf04bd"
server_key_version: "encryptionkey@0"
I20260925 02:49:58.928222 2852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:58.960759 2852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:58.965279 2852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:58.965587 2852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:58.966671 2852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:49:58.968139 2852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:49:58.968240 2852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:58.968391 2852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:49:58.968472 2852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:59.006278 2852 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:39109
I20260925 02:49:59.006372 2985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:39109 every 8 connection(s)
I20260925 02:49:59.007750 2852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:49:59.015084 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 2852
I20260925 02:49:59.015365 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:49:59.016076 2737 external_mini_cluster.cc:1428] Setting key 39312942383c38cbf98ac4f39867b8b2
Sep 25 02:49:59 dist-test-slave-g1xh krb5kdc[2744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:49:59.030798 2988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:59.010765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:55883 (local address 127.2.172.126:44921)
0925 02:49:59.011252 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:59.011256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:59.012512 (+ 1256us) server_negotiation.cc:407] Connection header received
0925 02:49:59.014666 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:59.014672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:59.014767 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:59.014899 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:59.016898 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:59.017834 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:59.019698 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:59.020111 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:59.020262 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:59.024485 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:59.024491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:59.024495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:59.024531 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:59.027182 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:59.028250 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:59.028261 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:59.028263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:59.028402 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:59.028904 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:59.028908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:59.028910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:59.029199 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:59.029356 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:59.030169 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 02:49:59.030405 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":257,"threads_started":1}
I20260925 02:49:59.031942 2989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:59.011444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:44921 (local address 127.2.172.65:55883)
0925 02:49:59.012266 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:59.012313 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:59.014335 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:59.015197 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:59.015210 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:59.015948 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:59.016664 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:59.016685 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:59.018080 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:59.018086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:59.019476 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:59.019495 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:59.020369 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:59.020378 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:59.020677 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:59.021519 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:59.021545 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:59.024202 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:59.027351 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:59.027361 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:59.027381 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:59.028097 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:59.028525 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:59.028530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:59.028534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:59.028771 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:59.029330 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:59.029338 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:59.029726 (+ 388us) client_negotiation.cc:769] Sending connection context
0925 02:49:59.030077 (+ 351us) client_negotiation.cc:240] Negotiation successful
0925 02:49:59.030644 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":312,"threads_started":1}
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
I20260925 02:49:59.033753 2986 heartbeater.cc:347] Connected to a master server at 127.2.172.126:44921
I20260925 02:49:59.034113 2986 heartbeater.cc:464] Registering TS with master...
I20260925 02:49:59.034972 2986 heartbeater.cc:511] Master 127.2.172.126:44921 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
I20260925 02:49:59.038749 2789 ts_manager.cc:194] Registered new tserver with Master: 2c462178941b4587a3abb6db6130f4f5 (127.2.172.65:39109)
I20260925 02:49:59.041322 2789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:55883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:49:59.063067 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:44921
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:59.426440 2992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:59.426728 2992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:59.426826 2992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:59.434306 2992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:59.434432 2992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:59.434551 2992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:49:59.444866 2992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:44921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:59.446197 2992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:59.447641 2992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:59.462949 2998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:59.462949 3000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:59.462989 2997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:59.464140 2992 server_base.cc:1034] running on GCE node
I20260925 02:49:59.464862 2992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:59.465927 2992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:59.467200 2992 hybrid_clock.cc:648] HybridClock initialized: now 1790304599467136 us; error 53 us; skew 500 ppm
I20260925 02:49:59.467624 2992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:59 dist-test-slave-g1xh krb5kdc[2744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:59.471391 2992 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:49:59.473299 2992 webserver.cc:460] Webserver started at http://127.2.172.66:38895/ using document root <none> and password file <none>
I20260925 02:49:59.473990 2992 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:59.474097 2992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:59.474365 2992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:59.477754 2992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "dd9382f451f748d58cb1cae127750d2f"
format_stamp: "Formatted at 2026-09-25 02:49:59 on dist-test-slave-g1xh"
server_key: "49c3b773363acbef4e39e6e3ca08bdc8"
server_key_iv: "8c36e6c3fbba273a24a804ecf5bed86e"
server_key_version: "encryptionkey@0"
I20260925 02:49:59.478582 2992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "dd9382f451f748d58cb1cae127750d2f"
format_stamp: "Formatted at 2026-09-25 02:49:59 on dist-test-slave-g1xh"
server_key: "49c3b773363acbef4e39e6e3ca08bdc8"
server_key_iv: "8c36e6c3fbba273a24a804ecf5bed86e"
server_key_version: "encryptionkey@0"
I20260925 02:49:59.484302 2992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:49:59.488260 3007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:59.489795 2992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:49:59.489956 2992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "dd9382f451f748d58cb1cae127750d2f"
format_stamp: "Formatted at 2026-09-25 02:49:59 on dist-test-slave-g1xh"
server_key: "49c3b773363acbef4e39e6e3ca08bdc8"
server_key_iv: "8c36e6c3fbba273a24a804ecf5bed86e"
server_key_version: "encryptionkey@0"
I20260925 02:49:59.490147 2992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:59.525815 2992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:59.530440 2992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:59.530704 2992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:59.531757 2992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:49:59.533234 2992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:49:59.533317 2992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:59.533422 2992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:49:59.533468 2992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:59.576213 2992 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:46577
I20260925 02:49:59.576247 3122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:46577 every 8 connection(s)
I20260925 02:49:59.577770 2992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:49:59.583297 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 2992
I20260925 02:49:59.583527 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:49:59.584127 2737 external_mini_cluster.cc:1428] Setting key 63e99d591c10e1c56413ccc9e02297e2
Sep 25 02:49:59 dist-test-slave-g1xh krb5kdc[2744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
I20260925 02:49:59.602576 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:59.580912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:36759 (local address 127.2.172.126:44921)
0925 02:49:59.581409 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:59.581414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:59.582558 (+ 1144us) server_negotiation.cc:407] Connection header received
0925 02:49:59.585161 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:59.585167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:59.585254 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:59.585363 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:59.587617 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:59.588346 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:59.590937 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:59.591371 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:59.591604 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:59.595878 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:59.595885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:59.595890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:59.595937 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:59.598806 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:59.599770 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:59.599779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:59.599783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:59.599964 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:59.600476 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:59.600483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:59.600489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:59.600895 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:59.601079 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:59.601844 (+ 765us) server_negotiation.cc:299] Negotiation successful
0925 02:49:59.602136 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":257,"threads_started":1}
I20260925 02:49:59.603740 3126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:59.581596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:44921 (local address 127.2.172.66:36759)
0925 02:49:59.582341 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:59.582381 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:59.584896 (+ 2515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:59.585736 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:59.585749 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:59.586449 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:59.587318 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:59.587348 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:59.588527 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:59.588536 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:59.590774 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:59.590794 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:59.591663 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:59.591673 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:59.591935 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:59.592695 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:59.592728 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:59.595582 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:59.599002 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:59.599009 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:59.599023 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:59.599637 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:59.600115 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:59.600119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:59.600122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:59.600327 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:59.601049 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:59.601056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:59.601401 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 02:49:59.601683 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 02:49:59.602108 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":303,"threads_started":1}
I20260925 02:49:59.605329 3123 heartbeater.cc:347] Connected to a master server at 127.2.172.126:44921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
I20260925 02:49:59.605713 3123 heartbeater.cc:464] Registering TS with master...
I20260925 02:49:59.606485 3123 heartbeater.cc:511] Master 127.2.172.126:44921 requested a full tablet report, sending...
I20260925 02:49:59.608034 2789 ts_manager.cc:194] Registered new tserver with Master: dd9382f451f748d58cb1cae127750d2f (127.2.172.66:46577)
I20260925 02:49:59.608912 2789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:36759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:49:59.630966 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:44921
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:59.997797 3129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:59.998103 3129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:59.998260 3129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:00.006001 3129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:00.006115 3129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:00.006327 3129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:50:00.016520 3129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:44921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:00.017874 3129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:00.019259 3129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:00.033072 3135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:00.033134 3134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:00.033138 3137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:00.034252 3129 server_base.cc:1034] running on GCE node
I20260925 02:50:00.034914 3129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:00.035938 3129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:00.037151 3129 hybrid_clock.cc:648] HybridClock initialized: now 1790304600037100 us; error 36 us; skew 500 ppm
I20260925 02:50:00.037473 3129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:00 dist-test-slave-g1xh krb5kdc[2744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:00.040730 3129 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:50:00.042148 3129 webserver.cc:460] Webserver started at http://127.2.172.67:41509/ using document root <none> and password file <none>
I20260925 02:50:00.042878 3129 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:00.042980 3129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:00.043355 3129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:00.045656 2986 heartbeater.cc:503] Master 127.2.172.126:44921 was elected leader, sending a full tablet report...
I20260925 02:50:00.046718 3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "38f49072ce52468baa31de52c5241566"
format_stamp: "Formatted at 2026-09-25 02:50:00 on dist-test-slave-g1xh"
server_key: "fd6a83ea5ba70d81d36305284ac5d36e"
server_key_iv: "7eb1377593c5bad78474439fab1523c8"
server_key_version: "encryptionkey@0"
I20260925 02:50:00.047749 3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "38f49072ce52468baa31de52c5241566"
format_stamp: "Formatted at 2026-09-25 02:50:00 on dist-test-slave-g1xh"
server_key: "fd6a83ea5ba70d81d36305284ac5d36e"
server_key_iv: "7eb1377593c5bad78474439fab1523c8"
server_key_version: "encryptionkey@0"
I20260925 02:50:00.053426 3129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:50:00.057390 3144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:00.058873 3129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:50:00.059060 3129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "38f49072ce52468baa31de52c5241566"
format_stamp: "Formatted at 2026-09-25 02:50:00 on dist-test-slave-g1xh"
server_key: "fd6a83ea5ba70d81d36305284ac5d36e"
server_key_iv: "7eb1377593c5bad78474439fab1523c8"
server_key_version: "encryptionkey@0"
I20260925 02:50:00.059271 3129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:00.096223 3129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:00.101438 3129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:00.101709 3129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:00.102735 3129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:00.104208 3129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:00.104310 3129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:00.104466 3129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:00.104547 3129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:00.140626 3129 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:44769
I20260925 02:50:00.140672 3259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:44769 every 8 connection(s)
I20260925 02:50:00.141975 3129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:00.150625 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 3129
I20260925 02:50:00.150820 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:00.151288 2737 external_mini_cluster.cc:1428] Setting key d740a9c0718d27abf9492f0260eff944
Sep 25 02:50:00 dist-test-slave-g1xh krb5kdc[2744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:00.164299 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:00.144891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:39837 (local address 127.2.172.126:44921)
0925 02:50:00.145441 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:00.145449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:00.146635 (+ 1186us) server_negotiation.cc:407] Connection header received
0925 02:50:00.148560 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:00.148565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:00.148642 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:00.148751 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:00.150327 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.151392 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.153603 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.154073 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.154235 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:00.157868 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:00.157874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:00.157878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:00.157912 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:00.160860 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:00.161659 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:00.161665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:00.161669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:00.161807 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:00.162267 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:00.162272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:00.162275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:00.162552 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:00.162731 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:00.163630 (+ 899us) server_negotiation.cc:299] Negotiation successful
0925 02:50:00.163891 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":248,"threads_started":1}
I20260925 02:50:00.165304 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:00.145656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:44921 (local address 127.2.172.67:39837)
0925 02:50:00.146422 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:00.146513 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:00.148257 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:00.148962 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:00.148973 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:00.149547 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:00.150121 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:00.150138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.151596 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.151602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:00.153458 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:00.153475 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.154346 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.154353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:00.154601 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:00.155280 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:00.155308 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:00.157640 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:00.161036 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:00.161046 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:00.161074 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:00.161513 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:00.161923 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:00.161928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:00.161932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:00.162141 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:00.162718 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:00.162726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:00.163195 (+ 469us) client_negotiation.cc:769] Sending connection context
0925 02:50:00.163474 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 02:50:00.164090 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":294,"threads_started":1}
I20260925 02:50:00.166989 3260 heartbeater.cc:347] Connected to a master server at 127.2.172.126:44921
I20260925 02:50:00.167318 3260 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:00.168147 3260 heartbeater.cc:511] Master 127.2.172.126:44921 requested a full tablet report, sending...
I20260925 02:50:00.169545 2789 ts_manager.cc:194] Registered new tserver with Master: 38f49072ce52468baa31de52c5241566 (127.2.172.67:44769)
I20260925 02:50:00.170432 2789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:39837
I20260925 02:50:00.180714 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:50:00 dist-test-slave-g1xh krb5kdc[2744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304600, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:50:00 dist-test-slave-g1xh krb5kdc[2744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304600, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:00.228880 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:00.212209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.2.172.65:39109)
0925 02:50:00.212980 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:00.212988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:00.213019 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:50:00.213178 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:00.213185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:00.213417 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:00.213591 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:00.215178 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.216371 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.217712 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.218289 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.218390 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:00.222532 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:00.222546 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:00.222561 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:00.222598 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:00.225831 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:00.226747 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:00.226755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:00.226757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:00.226888 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:00.227273 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:00.227276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:00.227278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:00.227733 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:00.227980 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:00.228230 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 02:50:00.228440 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":457,"threads_started":1}
W20260925 02:50:00.230372 2896 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52688
Sep 25 02:50:00 dist-test-slave-g1xh krb5kdc[2744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:50:00 dist-test-slave-g1xh krb5kdc[2744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:00.276393 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:00.262069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.2.172.65:39109)
0925 02:50:00.262319 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:00.262326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:00.262440 (+ 114us) server_negotiation.cc:407] Connection header received
0925 02:50:00.262660 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:00.262665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:00.262764 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:00.262852 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:00.264414 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.265118 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.266369 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.266789 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.266888 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:00.271219 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:00.271224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:00.271228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:00.271285 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:00.274040 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:00.274711 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:00.274715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:00.274717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:00.274845 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:00.275154 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:00.275157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:00.275159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:00.275421 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:00.275680 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:00.275914 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 02:50:00.276081 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 02:50:00.278347 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 2852
I20260925 02:50:00.292145 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:00.279465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.2.172.65:39109)
0925 02:50:00.279720 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:00.279729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:00.280047 (+ 318us) server_negotiation.cc:407] Connection header received
0925 02:50:00.280235 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:00.280242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:00.280310 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:00.280417 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:00.281802 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.282541 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.283883 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.284325 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.284470 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:00.286757 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:00.286762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:00.286765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:00.286796 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:00.289533 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:00.290209 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:00.290216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:00.290220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:00.290366 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:00.290720 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:00.290728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:00.290731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:00.290986 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:00.291244 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:00.291542 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 02:50:00.291793 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:50:00.293123 2980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:00.389734 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 2852
I20260925 02:50:00.410406 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 2992
Sep 25 02:50:00 dist-test-slave-g1xh krb5kdc[2744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:50:00.428277 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:00.411407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58320 (local address 127.2.172.66:46577)
0925 02:50:00.412043 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:00.412050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:00.412078 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:50:00.412152 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:00.412159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:00.412380 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:00.412523 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:00.413865 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.414916 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.416360 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.416935 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.417046 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:00.421363 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:00.421376 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:00.421392 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:00.421432 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:00.424675 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:00.425525 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:00.425534 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:00.425537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:00.425688 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:00.426271 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:00.426275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:00.426277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:00.426701 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:00.426942 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:00.427716 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 02:50:00.427961 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":371,"threads_started":1}
I20260925 02:50:00.429615 3118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:00.513384 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 2992
I20260925 02:50:00.533887 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 3129
Sep 25 02:50:00 dist-test-slave-g1xh krb5kdc[2744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:50:00.551520 3277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:00.534883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57226 (local address 127.2.172.67:44769)
0925 02:50:00.535722 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:00.535729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:00.535759 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:50:00.535821 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:00.535828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:00.536045 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:00.536245 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:00.537568 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.538620 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.540048 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:00.540646 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:00.540793 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:00.545285 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:00.545302 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:00.545318 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:00.545364 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:00.548582 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:00.549373 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:00.549381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:00.549383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:00.549529 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:00.549981 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:00.549984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:00.549987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:00.550353 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:00.550614 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:00.550977 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 02:50:00.551198 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":444,"threads_started":1}
I20260925 02:50:00.553237 3255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:00.638248 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 3129
I20260925 02:50:00.656704 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 2757
I20260925 02:50:00.658061 2821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:00.769368 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 2757
2026-09-25T02:50:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4419 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:00 dist-test-slave-g1xh krb5kdc[3282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:00 dist-test-slave-g1xh krb5kdc[3282](info): set up 2 sockets
Sep 25 02:50:00 dist-test-slave-g1xh krb5kdc[3282](info): commencing operation
krb5kdc: starting...
W20260925 02:50:01.905958 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:50:02 dist-test-slave-g1xh krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:50:02.103011 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:42943
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:39613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:42943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:02.436556 3295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:02.436859 3295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:02.436947 3295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:02.444420 3295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:02.444532 3295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:02.444586 3295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:02.444650 3295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:02.444695 3295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:02.454618 3295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:39613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:42943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:42943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:02.456065 3295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:02.457504 3295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:02.469942 3303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:02.470062 3301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:02.470468 3300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:02.470758 3295 server_base.cc:1034] running on GCE node
I20260925 02:50:02.471693 3295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:02.473281 3295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:02.474537 3295 hybrid_clock.cc:648] HybridClock initialized: now 1790304602474475 us; error 53 us; skew 500 ppm
I20260925 02:50:02.474939 3295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:02 dist-test-slave-g1xh krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304602, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:02.485872 3295 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:50:02.487859 3295 webserver.cc:460] Webserver started at http://127.2.172.126:44107/ using document root <none> and password file <none>
I20260925 02:50:02.488732 3295 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:02.488903 3295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:02.489264 3295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:02.492844 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "60ef60e2c8f74496b6cb08d96166ffe0"
format_stamp: "Formatted at 2026-09-25 02:50:02 on dist-test-slave-g1xh"
server_key: "474a0598626f47bb26e0503757228d33"
server_key_iv: "e51d3fd49fc2312669931172938a424f"
server_key_version: "encryptionkey@0"
I20260925 02:50:02.493741 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "60ef60e2c8f74496b6cb08d96166ffe0"
format_stamp: "Formatted at 2026-09-25 02:50:02 on dist-test-slave-g1xh"
server_key: "474a0598626f47bb26e0503757228d33"
server_key_iv: "e51d3fd49fc2312669931172938a424f"
server_key_version: "encryptionkey@0"
I20260925 02:50:02.500017 3295 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 02:50:02.504182 3310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:02.505848 3295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:50:02.506070 3295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "60ef60e2c8f74496b6cb08d96166ffe0"
format_stamp: "Formatted at 2026-09-25 02:50:02 on dist-test-slave-g1xh"
server_key: "474a0598626f47bb26e0503757228d33"
server_key_iv: "e51d3fd49fc2312669931172938a424f"
server_key_version: "encryptionkey@0"
I20260925 02:50:02.506310 3295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:02.531301 3295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:02.536216 3295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:02.536558 3295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:02.557947 3295 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:42943
I20260925 02:50:02.557952 3362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:42943 every 8 connection(s)
I20260925 02:50:02.559674 3295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:02.562830 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 3295
I20260925 02:50:02.563094 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:02.563958 2737 external_mini_cluster.cc:1428] Setting key 6d602fb248456d910cca7a1d7d08a719
I20260925 02:50:02.564616 3363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:02.580423 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0: Bootstrap starting.
Sep 25 02:50:02 dist-test-slave-g1xh krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:02.584758 3363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:02.586298 3363 log.cc:824] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:02.589541 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0: No bootstrap required, opened a new log
I20260925 02:50:02.591220 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:02.566231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.2.172.126:42943)
0925 02:50:02.567048 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:02.567061 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:02.567104 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:50:02.569572 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:02.569598 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:02.570032 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:02.570385 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:02.572277 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:02.573662 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:02.575656 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:02.576518 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:02.576696 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:02.582199 (+ 5503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:02.582214 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:02.582239 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:02.582301 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:02.586661 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:02.587609 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:02.587623 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:02.587630 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:02.587815 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:02.588223 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:02.588228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:02.588236 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:02.588791 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:02.589142 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:02.589632 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 02:50:02.590113 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":370,"threads_started":1}
I20260925 02:50:02.598491 3363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ef60e2c8f74496b6cb08d96166ffe0" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42943 } }
I20260925 02:50:02.598925 3363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:02.599038 3363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ef60e2c8f74496b6cb08d96166ffe0, State: Initialized, Role: FOLLOWER
I20260925 02:50:02.599697 3363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ef60e2c8f74496b6cb08d96166ffe0" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42943 } }
I20260925 02:50:02.599951 3363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:02.600104 3363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:02.600268 3363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:02.602476 3363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ef60e2c8f74496b6cb08d96166ffe0" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42943 } }
I20260925 02:50:02.603076 3363 leader_election.cc:302] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ef60e2c8f74496b6cb08d96166ffe0; no voters:
I20260925 02:50:02.603848 3363 leader_election.cc:288] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:02.603971 3368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:02.604684 3368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [term 1 LEADER]: Becoming Leader. State: Replica: 60ef60e2c8f74496b6cb08d96166ffe0, State: Running, Role: LEADER
I20260925 02:50:02.605343 3368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ef60e2c8f74496b6cb08d96166ffe0" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42943 } }
I20260925 02:50:02.606308 3363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:02.608518 3369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60ef60e2c8f74496b6cb08d96166ffe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ef60e2c8f74496b6cb08d96166ffe0" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42943 } } }
I20260925 02:50:02.608969 3369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:02.609395 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ef60e2c8f74496b6cb08d96166ffe0. Latest consensus state: current_term: 1 leader_uuid: "60ef60e2c8f74496b6cb08d96166ffe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ef60e2c8f74496b6cb08d96166ffe0" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42943 } } }
I20260925 02:50:02.609822 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:02.611970 3376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:02.617856 3376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:02.628804 3376 catalog_manager.cc:1347] Generated new cluster ID: 3c9917e3397a467bb85d31984c2f0cf4
I20260925 02:50:02.628952 3376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:02.669034 3376 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:02.671289 3376 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:02.694981 3376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60ef60e2c8f74496b6cb08d96166ffe0: Generated new TSK 0
I20260925 02:50:02.696161 3376 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:50:02.764065 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:42943
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:39613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:03.142937 3389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:03.143251 3389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:03.143350 3389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:03.150906 3389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:03.151044 3389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:03.151165 3389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:50:03.161655 3389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:39613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:42943
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:03.163002 3389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:03.164366 3389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:03.179284 3394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:03.179211 3397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:03.179263 3395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:03.180240 3389 server_base.cc:1034] running on GCE node
I20260925 02:50:03.180967 3389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:03.182020 3389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:03.183240 3389 hybrid_clock.cc:648] HybridClock initialized: now 1790304603183189 us; error 38 us; skew 500 ppm
I20260925 02:50:03.183652 3389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:03 dist-test-slave-g1xh krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:03.187163 3389 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:50:03.188843 3389 webserver.cc:460] Webserver started at http://127.2.172.65:40587/ using document root <none> and password file <none>
I20260925 02:50:03.189553 3389 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:03.189659 3389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:03.189934 3389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:03.193092 3389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "09e77b6a1db24a7698ca936b735b582f"
format_stamp: "Formatted at 2026-09-25 02:50:03 on dist-test-slave-g1xh"
server_key: "1c37754d81ed423389ea25a5815a3a70"
server_key_iv: "7d40290f00cbd348676884aea3f0b79a"
server_key_version: "encryptionkey@0"
I20260925 02:50:03.193878 3389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "09e77b6a1db24a7698ca936b735b582f"
format_stamp: "Formatted at 2026-09-25 02:50:03 on dist-test-slave-g1xh"
server_key: "1c37754d81ed423389ea25a5815a3a70"
server_key_iv: "7d40290f00cbd348676884aea3f0b79a"
server_key_version: "encryptionkey@0"
I20260925 02:50:03.199419 3389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:50:03.203145 3404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:03.204919 3389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:03.205147 3389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "09e77b6a1db24a7698ca936b735b582f"
format_stamp: "Formatted at 2026-09-25 02:50:03 on dist-test-slave-g1xh"
server_key: "1c37754d81ed423389ea25a5815a3a70"
server_key_iv: "7d40290f00cbd348676884aea3f0b79a"
server_key_version: "encryptionkey@0"
I20260925 02:50:03.205370 3389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:03.227315 3389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:03.232437 3389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:03.232815 3389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:03.234004 3389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:03.235666 3389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:03.235778 3389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:03.235982 3389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:03.236099 3389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:03.281479 3389 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:35889
I20260925 02:50:03.281534 3521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:35889 every 8 connection(s)
I20260925 02:50:03.282902 3389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:03.287653 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 3389
I20260925 02:50:03.287889 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:03.288710 2737 external_mini_cluster.cc:1428] Setting key 361d5f67abc76819a3c00f8fab70105a
Sep 25 02:50:03 dist-test-slave-g1xh krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
I20260925 02:50:03.309821 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:03.285767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:40551 (local address 127.2.172.126:42943)
0925 02:50:03.286237 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:03.286241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:03.287370 (+ 1129us) server_negotiation.cc:407] Connection header received
0925 02:50:03.290091 (+ 2721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:03.290098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:03.290195 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:03.290328 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:03.292679 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:03.293531 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:03.296220 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:03.296707 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:03.296874 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:03.301358 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:03.301368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:03.301376 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:03.301431 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:03.305619 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:03.306704 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:03.306741 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:03.306749 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:03.306944 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:03.307514 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:03.307520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:03.307524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:03.307999 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:03.308195 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:03.309012 (+ 817us) server_negotiation.cc:299] Negotiation successful
0925 02:50:03.309341 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":259,"threads_started":1}
I20260925 02:50:03.310868 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:03.286319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42943 (local address 127.2.172.65:40551)
0925 02:50:03.287131 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:03.287198 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:03.289758 (+ 2560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:03.290647 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:03.290665 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:03.291501 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:03.292446 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:03.292478 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:03.293729 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:03.293737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:03.296053 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:03.296072 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:03.296996 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:03.297006 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:03.297281 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:03.298055 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:03.298101 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:03.301054 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:03.305841 (+ 4787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:03.305850 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:03.305865 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:03.306523 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:03.307105 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:03.307113 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:03.307118 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:03.307329 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:03.308178 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:03.308185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:03.308560 (+ 375us) client_negotiation.cc:769] Sending connection context
0925 02:50:03.308816 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 02:50:03.309372 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":316,"threads_started":1}
I20260925 02:50:03.312693 3522 heartbeater.cc:347] Connected to a master server at 127.2.172.126:42943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
I20260925 02:50:03.313067 3522 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:03.313853 3522 heartbeater.cc:511] Master 127.2.172.126:42943 requested a full tablet report, sending...
I20260925 02:50:03.316079 3327 ts_manager.cc:194] Registered new tserver with Master: 09e77b6a1db24a7698ca936b735b582f (127.2.172.65:35889)
I20260925 02:50:03.317538 3327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:40551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:50:03.340255 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:42943
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:39613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:03.709640 3528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:03.710044 3528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:03.710248 3528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:03.718230 3528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:03.718418 3528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:03.718632 3528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:50:03.730463 3528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:39613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:42943
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:03.731972 3528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:03.733450 3528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:03.749284 3533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:03.749228 3536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:03.749210 3534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:03.750413 3528 server_base.cc:1034] running on GCE node
I20260925 02:50:03.751291 3528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:03.752512 3528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:03.753814 3528 hybrid_clock.cc:648] HybridClock initialized: now 1790304603753757 us; error 47 us; skew 500 ppm
I20260925 02:50:03.754383 3528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:03 dist-test-slave-g1xh krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:03.757968 3528 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:50:03.759896 3528 webserver.cc:460] Webserver started at http://127.2.172.66:37831/ using document root <none> and password file <none>
I20260925 02:50:03.760784 3528 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:03.761108 3528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:03.761521 3528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:03.764854 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "5fdb2a021e7e417eb93d6290ad8b7c54"
format_stamp: "Formatted at 2026-09-25 02:50:03 on dist-test-slave-g1xh"
server_key: "2ac7ecb745fe74ffdb77eab2446909d0"
server_key_iv: "af0638058ffae0c33d0f997a652c6de1"
server_key_version: "encryptionkey@0"
I20260925 02:50:03.765790 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "5fdb2a021e7e417eb93d6290ad8b7c54"
format_stamp: "Formatted at 2026-09-25 02:50:03 on dist-test-slave-g1xh"
server_key: "2ac7ecb745fe74ffdb77eab2446909d0"
server_key_iv: "af0638058ffae0c33d0f997a652c6de1"
server_key_version: "encryptionkey@0"
I20260925 02:50:03.771939 3528 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260925 02:50:03.776494 3543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:03.778218 3528 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:50:03.778438 3528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "5fdb2a021e7e417eb93d6290ad8b7c54"
format_stamp: "Formatted at 2026-09-25 02:50:03 on dist-test-slave-g1xh"
server_key: "2ac7ecb745fe74ffdb77eab2446909d0"
server_key_iv: "af0638058ffae0c33d0f997a652c6de1"
server_key_version: "encryptionkey@0"
I20260925 02:50:03.778693 3528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:03.801738 3528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:03.806640 3528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:03.806991 3528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:03.808211 3528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:03.809712 3528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:03.809844 3528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:03.809974 3528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:03.810038 3528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:03.845847 3528 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:45479
I20260925 02:50:03.845886 3657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:45479 every 8 connection(s)
I20260925 02:50:03.847247 3528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:03.852550 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 3528
I20260925 02:50:03.852761 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:03.853207 2737 external_mini_cluster.cc:1428] Setting key 00edc69d6fd45ed5f15dc0986e4323fa
Sep 25 02:50:03 dist-test-slave-g1xh krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:03.872176 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:03.849976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:52153 (local address 127.2.172.126:42943)
0925 02:50:03.850492 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:03.850499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:03.851534 (+ 1035us) server_negotiation.cc:407] Connection header received
0925 02:50:03.853694 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:03.853700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:03.853789 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:03.853912 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:03.855772 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:03.856951 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:03.858900 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:03.859739 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:03.859887 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:03.864871 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:03.864877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:03.864881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:03.864920 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:03.868448 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:03.869376 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:03.869385 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:03.869392 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:03.869581 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:03.870060 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:03.870067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:03.870073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:03.870458 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:03.870631 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:03.871381 (+ 750us) server_negotiation.cc:299] Negotiation successful
0925 02:50:03.871692 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":257,"threads_started":1}
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
I20260925 02:50:03.872984 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:03.850613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42943 (local address 127.2.172.66:52153)
0925 02:50:03.851349 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:03.851394 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:03.853341 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:03.854232 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:03.854241 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:03.854770 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:03.855496 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:03.855542 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:03.857184 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:03.857189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:03.858734 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:03.858748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:03.860008 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:03.860016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:03.860255 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:03.861140 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:03.861177 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:03.864559 (+ 3382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:03.868663 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:03.868671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:03.868687 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:03.869205 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:03.869712 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:03.869715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:03.869717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:03.869914 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:03.870602 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:03.870608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:03.870961 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 02:50:03.871214 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 02:50:03.871733 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":273,"threads_started":1}
I20260925 02:50:03.874727 3658 heartbeater.cc:347] Connected to a master server at 127.2.172.126:42943
I20260925 02:50:03.875121 3658 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
I20260925 02:50:03.876245 3658 heartbeater.cc:511] Master 127.2.172.126:42943 requested a full tablet report, sending...
I20260925 02:50:03.878018 3327 ts_manager.cc:194] Registered new tserver with Master: 5fdb2a021e7e417eb93d6290ad8b7c54 (127.2.172.66:45479)
I20260925 02:50:03.878912 3327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:52153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:50:03.900066 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:42943
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:39613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:04.251076 3664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:04.251396 3664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:04.251539 3664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:04.259155 3664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:04.259285 3664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:04.259485 3664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:50:04.270009 3664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:39613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:42943
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:04.271442 3664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:04.272934 3664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:04.287184 3669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:04.287150 3670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:04.288003 3672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:04.289081 3664 server_base.cc:1034] running on GCE node
I20260925 02:50:04.289757 3664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:04.290834 3664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:04.292104 3664 hybrid_clock.cc:648] HybridClock initialized: now 1790304604292053 us; error 41 us; skew 500 ppm
I20260925 02:50:04.292467 3664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:04 dist-test-slave-g1xh krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:04.295902 3664 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:50:04.297428 3664 webserver.cc:460] Webserver started at http://127.2.172.67:42709/ using document root <none> and password file <none>
I20260925 02:50:04.298135 3664 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:04.298239 3664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:04.298576 3664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:04.301647 3664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d"
format_stamp: "Formatted at 2026-09-25 02:50:04 on dist-test-slave-g1xh"
server_key: "33e0716107c1dcbdebef46097e9ad455"
server_key_iv: "71b8c87ccdf03a857ea96efdb34e9ebb"
server_key_version: "encryptionkey@0"
I20260925 02:50:04.302419 3664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d"
format_stamp: "Formatted at 2026-09-25 02:50:04 on dist-test-slave-g1xh"
server_key: "33e0716107c1dcbdebef46097e9ad455"
server_key_iv: "71b8c87ccdf03a857ea96efdb34e9ebb"
server_key_version: "encryptionkey@0"
I20260925 02:50:04.308844 3664 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260925 02:50:04.313190 3679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:04.314828 3664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:50:04.315040 3664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d"
format_stamp: "Formatted at 2026-09-25 02:50:04 on dist-test-slave-g1xh"
server_key: "33e0716107c1dcbdebef46097e9ad455"
server_key_iv: "71b8c87ccdf03a857ea96efdb34e9ebb"
server_key_version: "encryptionkey@0"
I20260925 02:50:04.315275 3664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:04.321430 3522 heartbeater.cc:503] Master 127.2.172.126:42943 was elected leader, sending a full tablet report...
I20260925 02:50:04.330430 3664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:04.335094 3664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:04.335418 3664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:04.336779 3664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:04.338297 3664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:04.338414 3664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:04.338564 3664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:04.338644 3664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:04.375626 3664 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:36847
I20260925 02:50:04.375669 3794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:36847 every 8 connection(s)
I20260925 02:50:04.376953 3664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:04.380419 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 3664
I20260925 02:50:04.380620 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:04.381254 2737 external_mini_cluster.cc:1428] Setting key 19ca5b4b2debf697c1c56c2354b0fe7f
Sep 25 02:50:04 dist-test-slave-g1xh krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:04.400408 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.379945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:35721 (local address 127.2.172.126:42943)
0925 02:50:04.380473 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:04.380479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:04.381922 (+ 1443us) server_negotiation.cc:407] Connection header received
0925 02:50:04.384383 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:04.384392 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:04.384512 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:04.384606 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:04.386273 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.387193 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.389028 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.389515 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.389630 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.393333 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:04.393338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:04.393341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:04.393378 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:04.396634 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:04.397732 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:04.397738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:04.397741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:04.397908 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:04.398486 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:04.398490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:04.398493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:04.398788 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:04.398914 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:04.399729 (+ 815us) server_negotiation.cc:299] Negotiation successful
0925 02:50:04.400095 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260925 02:50:04.401252 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.380715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42943 (local address 127.2.172.67:35721)
0925 02:50:04.381674 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:04.381714 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:04.384013 (+ 2299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:04.384889 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:04.384897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:04.385465 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:04.386078 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:04.386100 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.387368 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.387372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:04.388837 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:04.388851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.389767 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.389772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:04.389984 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.390732 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:04.390756 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:04.393075 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:04.396820 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:04.396829 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:04.396892 (+ 63us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:04.397560 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:04.398065 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:04.398069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:04.398073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:04.398305 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:04.398906 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:04.398912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:04.399274 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 02:50:04.399505 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 02:50:04.400037 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":362,"threads_started":1}
I20260925 02:50:04.402941 3795 heartbeater.cc:347] Connected to a master server at 127.2.172.126:42943
I20260925 02:50:04.403301 3795 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:04.404186 3795 heartbeater.cc:511] Master 127.2.172.126:42943 requested a full tablet report, sending...
I20260925 02:50:04.405725 3327 ts_manager.cc:194] Registered new tserver with Master: b5d54ccfddbc4defb3dbcfce1dc8b03d (127.2.172.67:36847)
I20260925 02:50:04.406811 3327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:35721
I20260925 02:50:04.411094 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:50:04.432929 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.418783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.2.172.126:42943)
0925 02:50:04.419018 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:04.419023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:04.419741 (+ 718us) server_negotiation.cc:407] Connection header received
0925 02:50:04.419981 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:04.419987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:04.420087 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:04.420202 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:04.421817 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.422576 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.423955 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.424374 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.424518 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.427082 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:04.427087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:04.427091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:04.427130 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:04.429818 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:04.430586 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:04.430592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:04.430596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:04.430736 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:04.431144 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:04.431151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:04.431154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:04.431515 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:04.431850 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:04.432306 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 02:50:04.432554 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 02:50:04.440822 3327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:04.449105 3327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:04.481156 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.464553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:45479 (local address 127.0.0.1:42330)
0925 02:50:04.470282 (+ 5729us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:04.470311 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:04.470550 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:04.472507 (+ 1957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:04.472516 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:04.472625 (+ 109us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:04.473715 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:04.473724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.475729 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.475734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:04.478286 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:04.478305 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.479919 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.479926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:04.480181 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.480271 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 02:50:04.480448 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 02:50:04.480705 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2812,"mutex_wait_us":299,"spinlock_wait_cycles":9472,"thread_start_us":4685,"threads_started":1}
I20260925 02:50:04.481534 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.464758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.2.172.66:45479)
0925 02:50:04.471620 (+ 6862us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:04.471630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:04.471654 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:50:04.471747 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:04.471753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:04.471997 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:04.472209 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:04.473973 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.475511 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.478536 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.479678 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.479866 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.480897 (+ 1031us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:04.481005 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 02:50:04.481210 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6646,"thread_start_us":6552,"threads_started":1}
I20260925 02:50:04.481739 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.464953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:35889 (local address 127.0.0.1:33026)
0925 02:50:04.469846 (+ 4893us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:04.469923 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:04.470858 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:04.472523 (+ 1665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:04.472528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:04.472574 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:04.473250 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:04.473264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.475209 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.475217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:04.478303 (+ 3086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:04.478313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.480749 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.480756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:04.480980 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.481022 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:50:04.481139 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 02:50:04.481281 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4536,"mutex_wait_us":173,"spinlock_wait_cycles":3456,"thread_start_us":4262,"threads_started":1}
I20260925 02:50:04.482546 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.465128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.2.172.65:35889)
0925 02:50:04.466993 (+ 1865us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:04.467008 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:04.470103 (+ 3095us) server_negotiation.cc:407] Connection header received
0925 02:50:04.471140 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:04.471155 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:04.471835 (+ 680us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:04.472212 (+ 377us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:04.473447 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.475037 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.478572 (+ 3535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.480124 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.480273 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.480563 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:04.482011 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0925 02:50:04.482228 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1588,"thread_start_us":1279,"threads_started":1}
I20260925 02:50:04.483173 3812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.466398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:36847 (local address 127.0.0.1:33316)
0925 02:50:04.471210 (+ 4812us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:04.471236 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:04.471400 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:04.472507 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:04.472516 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:04.472657 (+ 141us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:04.473710 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:04.473724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.475256 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.475262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:04.478221 (+ 2959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:04.478239 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.482252 (+ 4013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.482262 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:04.482473 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.482511 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:50:04.482630 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 02:50:04.482784 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4669,"mutex_wait_us":273,"spinlock_wait_cycles":205184,"thread_start_us":4517,"threads_started":1}
I20260925 02:50:04.484005 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.467033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.2.172.67:36847)
0925 02:50:04.470660 (+ 3627us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:04.470675 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:04.471580 (+ 905us) server_negotiation.cc:407] Connection header received
0925 02:50:04.471706 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:04.471716 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:04.471999 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:04.472209 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:04.473986 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.475053 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.478434 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.480124 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.480276 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.480572 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:04.483442 (+ 2870us) server_negotiation.cc:299] Negotiation successful
0925 02:50:04.483701 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3410,"thread_start_us":3234,"threads_started":1}
I20260925 02:50:04.486976 3455 tablet_service.cc:1467] Processing CreateTablet for tablet 69ad988228e34b789b9ddff475f70918 (DEFAULT_TABLE table=test-table [id=5b160d7f3d5446be99947df1d17b3404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:04.486976 3592 tablet_service.cc:1467] Processing CreateTablet for tablet 69ad988228e34b789b9ddff475f70918 (DEFAULT_TABLE table=test-table [id=5b160d7f3d5446be99947df1d17b3404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:04.488188 3729 tablet_service.cc:1467] Processing CreateTablet for tablet 69ad988228e34b789b9ddff475f70918 (DEFAULT_TABLE table=test-table [id=5b160d7f3d5446be99947df1d17b3404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:04.488991 3455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ad988228e34b789b9ddff475f70918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:04.489008 3592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ad988228e34b789b9ddff475f70918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:04.489907 3729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ad988228e34b789b9ddff475f70918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:04.501693 3814 tablet_bootstrap.cc:492] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f: Bootstrap starting.
I20260925 02:50:04.506111 3815 tablet_bootstrap.cc:492] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d: Bootstrap starting.
I20260925 02:50:04.507428 3814 tablet_bootstrap.cc:654] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:04.507489 3816 tablet_bootstrap.cc:492] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54: Bootstrap starting.
I20260925 02:50:04.509215 3814 log.cc:824] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:04.511073 3815 tablet_bootstrap.cc:654] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:04.511824 3816 tablet_bootstrap.cc:654] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:04.512352 3815 log.cc:824] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:04.513468 3816 log.cc:824] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:04.515904 3814 tablet_bootstrap.cc:492] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f: No bootstrap required, opened a new log
I20260925 02:50:04.516014 3815 tablet_bootstrap.cc:492] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d: No bootstrap required, opened a new log
I20260925 02:50:04.516259 3814 ts_tablet_manager.cc:1397] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260925 02:50:04.516309 3815 ts_tablet_manager.cc:1397] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 02:50:04.516650 3816 tablet_bootstrap.cc:492] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54: No bootstrap required, opened a new log
I20260925 02:50:04.516935 3816 ts_tablet_manager.cc:1397] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 02:50:04.524529 3814 raft_consensus.cc:348] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdb2a021e7e417eb93d6290ad8b7c54" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 45479 } } peers { permanent_uuid: "09e77b6a1db24a7698ca936b735b582f" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35889 } } peers { permanent_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36847 } }
I20260925 02:50:04.525023 3814 raft_consensus.cc:374] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:04.525127 3814 raft_consensus.cc:729] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e77b6a1db24a7698ca936b735b582f, State: Initialized, Role: FOLLOWER
I20260925 02:50:04.525810 3814 consensus_queue.cc:260] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdb2a021e7e417eb93d6290ad8b7c54" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 45479 } } peers { permanent_uuid: "09e77b6a1db24a7698ca936b735b582f" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35889 } } peers { permanent_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36847 } }
I20260925 02:50:04.527892 3814 ts_tablet_manager.cc:1428] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260925 02:50:04.528421 3816 raft_consensus.cc:348] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdb2a021e7e417eb93d6290ad8b7c54" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 45479 } } peers { permanent_uuid: "09e77b6a1db24a7698ca936b735b582f" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35889 } } peers { permanent_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36847 } }
I20260925 02:50:04.528941 3816 raft_consensus.cc:374] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:04.529179 3816 raft_consensus.cc:729] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fdb2a021e7e417eb93d6290ad8b7c54, State: Initialized, Role: FOLLOWER
I20260925 02:50:04.528744 3815 raft_consensus.cc:348] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdb2a021e7e417eb93d6290ad8b7c54" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 45479 } } peers { permanent_uuid: "09e77b6a1db24a7698ca936b735b582f" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35889 } } peers { permanent_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36847 } }
I20260925 02:50:04.529361 3815 raft_consensus.cc:374] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:04.529469 3815 raft_consensus.cc:729] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d54ccfddbc4defb3dbcfce1dc8b03d, State: Initialized, Role: FOLLOWER
I20260925 02:50:04.529910 3816 consensus_queue.cc:260] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdb2a021e7e417eb93d6290ad8b7c54" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 45479 } } peers { permanent_uuid: "09e77b6a1db24a7698ca936b735b582f" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35889 } } peers { permanent_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36847 } }
I20260925 02:50:04.530064 3815 consensus_queue.cc:260] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdb2a021e7e417eb93d6290ad8b7c54" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 45479 } } peers { permanent_uuid: "09e77b6a1db24a7698ca936b735b582f" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35889 } } peers { permanent_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36847 } }
W20260925 02:50:04.535640 3523 tablet.cc:2369] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:04.537007 3795 heartbeater.cc:503] Master 127.2.172.126:42943 was elected leader, sending a full tablet report...
I20260925 02:50:04.537015 3658 heartbeater.cc:503] Master 127.2.172.126:42943 was elected leader, sending a full tablet report...
I20260925 02:50:04.537340 3816 ts_tablet_manager.cc:1428] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54: Time spent starting tablet: real 0.020s user 0.010s sys 0.011s
I20260925 02:50:04.537340 3815 ts_tablet_manager.cc:1428] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d: Time spent starting tablet: real 0.021s user 0.016s sys 0.003s
W20260925 02:50:04.599345 3659 tablet.cc:2369] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:50:04.628824 3796 tablet.cc:2369] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:04.731720 3822 raft_consensus.cc:482] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:04.732167 3822 raft_consensus.cc:504] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdb2a021e7e417eb93d6290ad8b7c54" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 45479 } } peers { permanent_uuid: "09e77b6a1db24a7698ca936b735b582f" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35889 } } peers { permanent_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36847 } }
I20260925 02:50:04.734148 3822 leader_election.cc:288] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fdb2a021e7e417eb93d6290ad8b7c54 (127.2.172.66:45479), 09e77b6a1db24a7698ca936b735b582f (127.2.172.65:35889)
I20260925 02:50:04.741251 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.734486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:35889 (local address 127.2.172.67:47443)
0925 02:50:04.734922 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:04.734950 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:04.735143 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:04.735668 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:04.735677 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:04.735716 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:04.736041 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:04.736055 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.737277 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.737283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:04.738970 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:04.738984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.740361 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.740383 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:04.740598 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.740641 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 02:50:04.740756 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:50:04.740922 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260925 02:50:04.741236 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.734735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:47443 (local address 127.2.172.65:35889)
0925 02:50:04.734960 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:04.734965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:04.735031 (+ 66us) server_negotiation.cc:407] Connection header received
0925 02:50:04.735327 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:04.735333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:04.735414 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:04.735539 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:04.736218 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.737128 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.739148 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.740189 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.740318 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.740389 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:04.740789 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 02:50:04.740927 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 02:50:04.741827 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.734739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:56739 (local address 127.2.172.66:45479)
0925 02:50:04.734952 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:04.734959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:04.735644 (+ 685us) server_negotiation.cc:407] Connection header received
0925 02:50:04.735842 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:04.735848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:04.735935 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:04.736055 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:04.736720 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.737663 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.739682 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.740819 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.740985 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.741059 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:04.741448 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 02:50:04.741642 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:50:04.742493 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.734486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:45479 (local address 127.2.172.67:56739)
0925 02:50:04.735510 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:04.735529 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:04.735670 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:04.736088 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:04.736092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:04.736156 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:04.736548 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:04.736561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.737820 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.737824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:04.739449 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:04.739474 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.741035 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.741042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:04.741215 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.741260 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:50:04.742039 (+ 779us) client_negotiation.cc:240] Negotiation successful
0925 02:50:04.742191 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":41,"thread_start_us":580,"threads_started":1}
I20260925 02:50:04.742493 3475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ad988228e34b789b9ddff475f70918" candidate_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e77b6a1db24a7698ca936b735b582f" is_pre_election: true
I20260925 02:50:04.743347 3475 raft_consensus.cc:2454] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5d54ccfddbc4defb3dbcfce1dc8b03d in term 0.
I20260925 02:50:04.743794 3612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ad988228e34b789b9ddff475f70918" candidate_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fdb2a021e7e417eb93d6290ad8b7c54" is_pre_election: true
I20260925 02:50:04.744400 3680 leader_election.cc:302] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09e77b6a1db24a7698ca936b735b582f, b5d54ccfddbc4defb3dbcfce1dc8b03d; no voters:
I20260925 02:50:04.744517 3612 raft_consensus.cc:2454] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5d54ccfddbc4defb3dbcfce1dc8b03d in term 0.
I20260925 02:50:04.744966 3822 raft_consensus.cc:2781] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:04.745146 3822 raft_consensus.cc:482] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:04.745275 3822 raft_consensus.cc:3037] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:04.747812 3822 raft_consensus.cc:504] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdb2a021e7e417eb93d6290ad8b7c54" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 45479 } } peers { permanent_uuid: "09e77b6a1db24a7698ca936b735b582f" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35889 } } peers { permanent_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36847 } }
I20260925 02:50:04.748548 3822 leader_election.cc:288] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [CANDIDATE]: Term 1 election: Requested vote from peers 5fdb2a021e7e417eb93d6290ad8b7c54 (127.2.172.66:45479), 09e77b6a1db24a7698ca936b735b582f (127.2.172.65:35889)
I20260925 02:50:04.749153 3475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ad988228e34b789b9ddff475f70918" candidate_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e77b6a1db24a7698ca936b735b582f"
I20260925 02:50:04.749208 3612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ad988228e34b789b9ddff475f70918" candidate_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fdb2a021e7e417eb93d6290ad8b7c54"
I20260925 02:50:04.749420 3612 raft_consensus.cc:3037] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:04.749418 3475 raft_consensus.cc:3037] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:04.753104 3612 raft_consensus.cc:2454] T 69ad988228e34b789b9ddff475f70918 P 5fdb2a021e7e417eb93d6290ad8b7c54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5d54ccfddbc4defb3dbcfce1dc8b03d in term 1.
I20260925 02:50:04.753104 3475 raft_consensus.cc:2454] T 69ad988228e34b789b9ddff475f70918 P 09e77b6a1db24a7698ca936b735b582f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5d54ccfddbc4defb3dbcfce1dc8b03d in term 1.
I20260925 02:50:04.753774 3684 leader_election.cc:302] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fdb2a021e7e417eb93d6290ad8b7c54, b5d54ccfddbc4defb3dbcfce1dc8b03d; no voters:
I20260925 02:50:04.754228 3822 raft_consensus.cc:2781] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:04.754773 3822 raft_consensus.cc:686] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [term 1 LEADER]: Becoming Leader. State: Replica: b5d54ccfddbc4defb3dbcfce1dc8b03d, State: Running, Role: LEADER
I20260925 02:50:04.755371 3822 consensus_queue.cc:237] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdb2a021e7e417eb93d6290ad8b7c54" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 45479 } } peers { permanent_uuid: "09e77b6a1db24a7698ca936b735b582f" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35889 } } peers { permanent_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36847 } }
I20260925 02:50:04.760852 3327 catalog_manager.cc:5495] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d reported cstate change: term changed from 0 to 1, leader changed from <none> to b5d54ccfddbc4defb3dbcfce1dc8b03d (127.2.172.67). New cstate: current_term: 1 leader_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdb2a021e7e417eb93d6290ad8b7c54" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 45479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09e77b6a1db24a7698ca936b735b582f" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5d54ccfddbc4defb3dbcfce1dc8b03d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36847 } health_report { overall_health: HEALTHY } } }
Sep 25 02:50:04 dist-test-slave-g1xh krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:04.853324 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.836912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.2.172.65:35889)
0925 02:50:04.837183 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:04.837192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:04.837704 (+ 512us) server_negotiation.cc:407] Connection header received
0925 02:50:04.837840 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:04.837844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:04.837922 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:04.838036 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:04.839793 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.840576 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.841900 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.842351 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.842480 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.846714 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:04.846732 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:04.846747 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:04.846789 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:04.849939 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:04.851020 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:04.851028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:04.851031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:04.851176 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:04.851608 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:04.851615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:04.851623 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:04.852014 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:04.852343 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:04.852707 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 02:50:04.852953 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
Sep 25 02:50:04 dist-test-slave-g1xh krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304604, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:50:04 dist-test-slave-g1xh krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304604, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:04.903765 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.889084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.2.172.65:35889)
0925 02:50:04.889332 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:04.889338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:04.889609 (+ 271us) server_negotiation.cc:407] Connection header received
0925 02:50:04.889837 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:04.889842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:04.889908 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:04.890021 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:04.891583 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.892345 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.893617 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.894043 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.894184 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.898195 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:04.898200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:04.898203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:04.898242 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:04.901023 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:04.901713 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:04.901724 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:04.901726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:04.901869 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:04.902304 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:04.902310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:04.902313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:04.902598 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:04.902859 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:04.903179 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 02:50:04.903406 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 02:50:04 dist-test-slave-g1xh krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:50:04 dist-test-slave-g1xh krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:04.952924 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.938010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.2.172.65:35889)
0925 02:50:04.938242 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:04.938247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:04.938268 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:50:04.938442 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:04.938447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:04.938514 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:04.938605 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:04.939951 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.940672 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.941927 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.942350 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.942460 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.946690 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:04.946696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:04.946699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:04.946756 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:04.949762 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:04.950580 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:04.950584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:04.950587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:04.950724 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:04.951180 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:04.951185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:04.951187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:04.951691 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:04.951914 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:04.952238 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 02:50:04.952494 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260925 02:50:04.954980 3434 tablet_service.cc:2908] Rejecting scan request for tablet 69ad988228e34b789b9ddff475f70918: Uninitialized: clean time has not yet been initialized
I20260925 02:50:04.959736 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 3389
I20260925 02:50:04.974511 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:04.960654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33056 (local address 127.2.172.65:35889)
0925 02:50:04.960850 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:04.960855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:04.961377 (+ 522us) server_negotiation.cc:407] Connection header received
0925 02:50:04.961470 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:04.961475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:04.961573 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:04.961698 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:04.963155 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.964030 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.965408 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:04.965904 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:04.966040 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:04.968685 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:04.968691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:04.968695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:04.968735 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:04.971641 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:04.972441 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:04.972447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:04.972450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:04.972593 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:04.973019 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:04.973023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:04.973026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:04.973369 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:04.973640 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:04.973877 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 02:50:04.974090 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:50:04.975222 3517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:05.069242 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 3389
I20260925 02:50:05.089064 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 3528
Sep 25 02:50:05 dist-test-slave-g1xh krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:50:05.104966 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:05.089902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.2.172.66:45479)
0925 02:50:05.090133 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:05.090140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:05.090162 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:50:05.090227 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:05.090231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:05.090300 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:05.090470 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:05.091716 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:05.092489 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:05.093848 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:05.094261 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:05.094366 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:05.098796 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:05.098814 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:05.098830 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:05.098871 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:05.101988 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:05.102706 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:05.102714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:05.102716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:05.102857 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:05.103263 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:05.103267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:05.103269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:05.103838 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:05.104078 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:05.104353 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 02:50:05.104583 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260925 02:50:05.105787 3652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:05.144698 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:05.144172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:35889 (local address 127.2.172.67:42209)
0925 02:50:05.144360 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:05.144552 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 02:50:05.144959 3680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.172.65:35889: connect: Connection refused (error 111)
I20260925 02:50:05.146546 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:05.146175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:35889 (local address 127.2.172.67:57711)
0925 02:50:05.146323 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:05.146410 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:50:05.146759 3680 consensus_peers.cc:487] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d -> Peer 09e77b6a1db24a7698ca936b735b582f (127.2.172.65:35889): Couldn't send request to peer 09e77b6a1db24a7698ca936b735b582f. Status: Network error: Client connection negotiation failed: client connection to 127.2.172.65:35889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:50:05.194038 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 3528
I20260925 02:50:05.210295 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 3664
I20260925 02:50:05.211411 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:05.210734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:45479 (local address 127.2.172.67:60145)
0925 02:50:05.210966 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:05.211118 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.66:45479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 02:50:05.211772 3684 consensus_peers.cc:487] T 69ad988228e34b789b9ddff475f70918 P b5d54ccfddbc4defb3dbcfce1dc8b03d -> Peer 5fdb2a021e7e417eb93d6290ad8b7c54 (127.2.172.66:45479): Couldn't send request to peer 5fdb2a021e7e417eb93d6290ad8b7c54. Status: Network error: Client connection negotiation failed: client connection to 127.2.172.66:45479: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 25 02:50:05 dist-test-slave-g1xh krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:50:05.226670 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:05.211135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.2.172.67:36847)
0925 02:50:05.211832 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:05.211839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:05.211861 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:50:05.211955 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:05.211966 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:05.212026 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:05.212117 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:05.213344 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:05.214069 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:05.215314 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:05.215858 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:05.215981 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:05.220338 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:05.220354 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:05.220383 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:05.220455 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:05.223693 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:05.224527 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:05.224543 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:05.224545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:05.224706 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:05.225117 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:05.225121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:05.225123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:05.225497 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:05.225721 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:05.226005 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 02:50:05.226192 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":305,"threads_started":1}
I20260925 02:50:05.227408 3790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:05.318364 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 3664
I20260925 02:50:05.336658 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 3295
I20260925 02:50:05.337958 3358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:05.451216 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 3295
2026-09-25T02:50:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4678 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:05 dist-test-slave-g1xh krb5kdc[3846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:05 dist-test-slave-g1xh krb5kdc[3846](info): set up 2 sockets
Sep 25 02:50:05 dist-test-slave-g1xh krb5kdc[3846](info): commencing operation
krb5kdc: starting...
W20260925 02:50:06.554214 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:50:06 dist-test-slave-g1xh krb5kdc[3846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:50:06.760156 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:37161
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:37161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:07.133628 3859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:07.134054 3859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:07.134227 3859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:07.142464 3859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:07.142638 3859 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:50:07.142693 3859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:07.142809 3859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:07.142918 3859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:07.143050 3859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:07.155706 3859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:37161
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:37161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:07.157778 3859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:07.159890 3859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:07.175974 3864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:07.175902 3865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:07.175933 3867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:07.176400 3859 server_base.cc:1034] running on GCE node
I20260925 02:50:07.177759 3859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:07.179924 3859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:07.181257 3859 hybrid_clock.cc:648] HybridClock initialized: now 1790304607181166 us; error 56 us; skew 500 ppm
I20260925 02:50:07.181689 3859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:07 dist-test-slave-g1xh krb5kdc[3846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304607, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:07.186088 3859 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:50:07.188148 3859 webserver.cc:460] Webserver started at http://127.2.172.126:41193/ using document root <none> and password file <none>
I20260925 02:50:07.189100 3859 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:07.189213 3859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:07.189570 3859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:07.193673 3859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "3dafc9a5e35e4bbb86b0dc74e782eef7"
format_stamp: "Formatted at 2026-09-25 02:50:07 on dist-test-slave-g1xh"
server_key: "85a4bfe06d26ddac804a0e509693f278"
server_key_iv: "4f59c7de123ee2e77c63098a21483a3e"
server_key_version: "encryptionkey@0"
I20260925 02:50:07.194648 3859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "3dafc9a5e35e4bbb86b0dc74e782eef7"
format_stamp: "Formatted at 2026-09-25 02:50:07 on dist-test-slave-g1xh"
server_key: "85a4bfe06d26ddac804a0e509693f278"
server_key_iv: "4f59c7de123ee2e77c63098a21483a3e"
server_key_version: "encryptionkey@0"
I20260925 02:50:07.201649 3859 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260925 02:50:07.206302 3874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:07.208408 3859 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 02:50:07.208611 3859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "3dafc9a5e35e4bbb86b0dc74e782eef7"
format_stamp: "Formatted at 2026-09-25 02:50:07 on dist-test-slave-g1xh"
server_key: "85a4bfe06d26ddac804a0e509693f278"
server_key_iv: "4f59c7de123ee2e77c63098a21483a3e"
server_key_version: "encryptionkey@0"
I20260925 02:50:07.208859 3859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:07.232587 3859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:07.238312 3859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:07.238595 3859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:07.266718 3859 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:37161
I20260925 02:50:07.266762 3936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:37161 every 8 connection(s)
I20260925 02:50:07.268664 3859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:07.273505 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 3859
I20260925 02:50:07.273838 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:07.274518 3937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:07.274623 2737 external_mini_cluster.cc:1428] Setting key af8e95ca470cf786aa60247abcb9d852
I20260925 02:50:07.294528 3937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7: Bootstrap starting.
Sep 25 02:50:07 dist-test-slave-g1xh krb5kdc[3846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:07.300388 3937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:07.302402 3937 log.cc:824] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:07.307679 3937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7: No bootstrap required, opened a new log
I20260925 02:50:07.309379 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:07.277576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.2.172.126:37161)
0925 02:50:07.278664 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:07.278686 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:07.278762 (+ 76us) server_negotiation.cc:407] Connection header received
0925 02:50:07.281715 (+ 2953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:07.281770 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:07.282658 (+ 888us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:07.283401 (+ 743us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:07.285388 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:07.288034 (+ 2646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:07.289689 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:07.291135 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:07.291422 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:07.297153 (+ 5731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:07.297169 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:07.297207 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:07.297327 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:07.302399 (+ 5072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:07.303880 (+ 1481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:07.303890 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:07.303910 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:07.304155 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:07.304696 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:07.304702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:07.304705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:07.305598 (+ 893us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:07.305996 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:07.306594 (+ 598us) server_negotiation.cc:299] Negotiation successful
0925 02:50:07.307320 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":456,"threads_started":1}
I20260925 02:50:07.318800 3937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dafc9a5e35e4bbb86b0dc74e782eef7" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 37161 } }
I20260925 02:50:07.319465 3937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:07.319640 3937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dafc9a5e35e4bbb86b0dc74e782eef7, State: Initialized, Role: FOLLOWER
I20260925 02:50:07.320533 3937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dafc9a5e35e4bbb86b0dc74e782eef7" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 37161 } }
I20260925 02:50:07.320848 3937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:07.321015 3937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:07.321228 3937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:07.324074 3937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dafc9a5e35e4bbb86b0dc74e782eef7" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 37161 } }
I20260925 02:50:07.324922 3937 leader_election.cc:302] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dafc9a5e35e4bbb86b0dc74e782eef7; no voters:
I20260925 02:50:07.325747 3937 leader_election.cc:288] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:07.325970 3942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:07.326979 3942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [term 1 LEADER]: Becoming Leader. State: Replica: 3dafc9a5e35e4bbb86b0dc74e782eef7, State: Running, Role: LEADER
I20260925 02:50:07.327845 3942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dafc9a5e35e4bbb86b0dc74e782eef7" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 37161 } }
I20260925 02:50:07.328879 3937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:07.332309 3943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dafc9a5e35e4bbb86b0dc74e782eef7. Latest consensus state: current_term: 1 leader_uuid: "3dafc9a5e35e4bbb86b0dc74e782eef7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dafc9a5e35e4bbb86b0dc74e782eef7" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 37161 } } }
I20260925 02:50:07.333187 3943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:07.332342 3945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dafc9a5e35e4bbb86b0dc74e782eef7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dafc9a5e35e4bbb86b0dc74e782eef7" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 37161 } } }
I20260925 02:50:07.333513 3945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:07.339416 3951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:07.347055 3951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:07.360240 3951 catalog_manager.cc:1347] Generated new cluster ID: ba9f848d9159402b817c5483a153223a
I20260925 02:50:07.360478 3951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:07.372102 3951 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:07.374364 3951 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:07.389479 3951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7: Generated new TSK 0
I20260925 02:50:07.390782 3951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:50:07.456640 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:37161
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:07.904796 3964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:07.905202 3964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:07.905377 3964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:07.913784 3964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:07.913992 3964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:07.914175 3964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:50:07.914312 3964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:50:07.926298 3964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.172.126:37161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:07.928403 3964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:07.930550 3964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:07.948396 3970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:07.948355 3969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:07.948367 3972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:07.949172 3964 server_base.cc:1034] running on GCE node
I20260925 02:50:07.950335 3964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:07.951664 3964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:07.952956 3964 hybrid_clock.cc:648] HybridClock initialized: now 1790304607952875 us; error 54 us; skew 500 ppm
I20260925 02:50:07.953418 3964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:07 dist-test-slave-g1xh krb5kdc[3846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304607, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:07.962977 3964 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:50:07.965348 3964 webserver.cc:460] Webserver started at http://127.2.172.65:35887/ using document root <none> and password file <none>
I20260925 02:50:07.966379 3964 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:07.966542 3964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:07.966979 3964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:07.971271 3964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "fac5091b06704f17b5fe94b423a033b4"
format_stamp: "Formatted at 2026-09-25 02:50:07 on dist-test-slave-g1xh"
server_key: "8feba8517b724cfa72b38d61e9990103"
server_key_iv: "cb347ddc81e97d6367d0b1e6e6b85613"
server_key_version: "encryptionkey@0"
I20260925 02:50:07.972404 3964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "fac5091b06704f17b5fe94b423a033b4"
format_stamp: "Formatted at 2026-09-25 02:50:07 on dist-test-slave-g1xh"
server_key: "8feba8517b724cfa72b38d61e9990103"
server_key_iv: "cb347ddc81e97d6367d0b1e6e6b85613"
server_key_version: "encryptionkey@0"
I20260925 02:50:07.979316 3964 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 02:50:07.984087 3979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:07.986099 3964 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:50:07.986357 3964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "fac5091b06704f17b5fe94b423a033b4"
format_stamp: "Formatted at 2026-09-25 02:50:07 on dist-test-slave-g1xh"
server_key: "8feba8517b724cfa72b38d61e9990103"
server_key_iv: "cb347ddc81e97d6367d0b1e6e6b85613"
server_key_version: "encryptionkey@0"
I20260925 02:50:07.986670 3964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:08.003711 3964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:08.009115 3964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:08.009497 3964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:08.013065 3964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:08.013206 3964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:08.013379 3964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:08.013460 3964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:50:08 dist-test-slave-g1xh krb5kdc[3846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304607, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:08.043195 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:08.014059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:47699 (local address 127.2.172.126:37161)
0925 02:50:08.014642 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:08.014648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:08.016339 (+ 1691us) server_negotiation.cc:407] Connection header received
0925 02:50:08.019157 (+ 2818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:08.019165 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:08.019302 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:08.019431 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:08.023636 (+ 4205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.024792 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.028492 (+ 3700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.029130 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.029292 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:08.035217 (+ 5925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:08.035223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:08.035227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:08.035275 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:08.038902 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:08.040202 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:08.040207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:08.040209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:08.040384 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:08.040940 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:08.040944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:08.040947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:08.041279 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:08.041462 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:08.042465 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0925 02:50:08.042746 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":332,"threads_started":1}
I20260925 02:50:08.045363 3989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:08.015033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:37161 (local address 127.2.172.65:47699)
0925 02:50:08.016121 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:08.016179 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:08.018646 (+ 2467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:08.020298 (+ 1652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:08.020318 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:08.021623 (+ 1305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:08.023265 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:08.023326 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.025011 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.025025 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:08.028244 (+ 3219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:08.028273 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.029450 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.029459 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:08.029903 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:08.031049 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:08.031104 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:08.034862 (+ 3758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:08.039089 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:08.039108 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:08.039171 (+ 63us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:08.040035 (+ 864us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:08.040521 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:08.040526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:08.040530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:08.040794 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:08.041452 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:08.041462 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:08.042002 (+ 540us) client_negotiation.cc:769] Sending connection context
0925 02:50:08.042343 (+ 341us) client_negotiation.cc:240] Negotiation successful
0925 02:50:08.043147 (+ 804us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":508,"threads_started":1}
I20260925 02:50:08.072798 4068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:08.054820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:37161 (local address 127.2.172.65:56573)
0925 02:50:08.055657 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:08.055684 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:08.055947 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:08.056396 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:08.056401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:08.056987 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:08.057354 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:08.057365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.058527 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.058532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:08.060108 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:08.060129 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.061372 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.061386 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:08.061601 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:08.062503 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:08.062542 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:08.064374 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:08.068830 (+ 4456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:08.068839 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:08.068851 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:08.069799 (+ 948us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:08.070310 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:08.070319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:08.070325 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:08.070558 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:08.071635 (+ 1077us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:08.071641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:08.071891 (+ 250us) client_negotiation.cc:769] Sending connection context
0925 02:50:08.072027 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 02:50:08.072366 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":472,"threads_started":1}
I20260925 02:50:08.073194 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:08.054249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:56573 (local address 127.2.172.126:37161)
0925 02:50:08.054562 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:08.054573 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:08.055832 (+ 1259us) server_negotiation.cc:407] Connection header received
0925 02:50:08.056130 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:08.056136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:08.056223 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:08.056340 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:08.057537 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.058362 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.060294 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.061050 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.061237 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:08.064688 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:08.064696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:08.064704 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:08.064762 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:08.068588 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:08.070001 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:08.070006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:08.070008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:08.070172 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:08.070762 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:08.070771 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:08.070778 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:08.071415 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:08.071694 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:08.072238 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 02:50:08.072627 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260925 02:50:08.079206 3964 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:43533
I20260925 02:50:08.079279 4100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:43533 every 8 connection(s)
I20260925 02:50:08.080968 3964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:08.084324 4101 heartbeater.cc:347] Connected to a master server at 127.2.172.126:37161
I20260925 02:50:08.084749 4101 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:08.085819 4101 heartbeater.cc:511] Master 127.2.172.126:37161 requested a full tablet report, sending...
I20260925 02:50:08.089216 3892 ts_manager.cc:194] Registered new tserver with Master: fac5091b06704f17b5fe94b423a033b4 (127.2.172.65:43533)
I20260925 02:50:08.090075 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 3964
I20260925 02:50:08.090356 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:08.091310 2737 external_mini_cluster.cc:1428] Setting key a5c1827b515866d05899a74bc3b32b29
I20260925 02:50:08.092073 3892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:47699
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:50:08.143989 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:37161
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:08.566550 4105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:08.567020 4105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:08.567221 4105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:08.575817 4105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:08.576020 4105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:08.576190 4105 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:50:08.576342 4105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:50:08.588583 4105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.172.126:37161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:08.590569 4105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:08.592669 4105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:08.609220 4112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:08.609200 4114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:08.609244 4111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:08.610303 4105 server_base.cc:1034] running on GCE node
I20260925 02:50:08.611137 4105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:08.612385 4105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:08.613628 4105 hybrid_clock.cc:648] HybridClock initialized: now 1790304608613552 us; error 39 us; skew 500 ppm
I20260925 02:50:08.614076 4105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:08 dist-test-slave-g1xh krb5kdc[3846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304608, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:08.618291 4105 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:50:08.620468 4105 webserver.cc:460] Webserver started at http://127.2.172.66:44847/ using document root <none> and password file <none>
I20260925 02:50:08.621484 4105 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:08.621644 4105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:08.622081 4105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:08.626114 4105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "86514a238b3e4b299c385f16229980b6"
format_stamp: "Formatted at 2026-09-25 02:50:08 on dist-test-slave-g1xh"
server_key: "2c2144d2c78a298e7fa53c5c1652ada6"
server_key_iv: "edb1f89f9b77159550001a61e0cbc2df"
server_key_version: "encryptionkey@0"
I20260925 02:50:08.627202 4105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "86514a238b3e4b299c385f16229980b6"
format_stamp: "Formatted at 2026-09-25 02:50:08 on dist-test-slave-g1xh"
server_key: "2c2144d2c78a298e7fa53c5c1652ada6"
server_key_iv: "edb1f89f9b77159550001a61e0cbc2df"
server_key_version: "encryptionkey@0"
I20260925 02:50:08.634625 4105 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260925 02:50:08.639541 4121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:08.641631 4105 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:50:08.641891 4105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "86514a238b3e4b299c385f16229980b6"
format_stamp: "Formatted at 2026-09-25 02:50:08 on dist-test-slave-g1xh"
server_key: "2c2144d2c78a298e7fa53c5c1652ada6"
server_key_iv: "edb1f89f9b77159550001a61e0cbc2df"
server_key_version: "encryptionkey@0"
I20260925 02:50:08.642167 4105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:08.665375 4105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:08.671130 4105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:08.671479 4105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:08.675398 4105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:08.675539 4105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:08.675747 4105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:08.675843 4105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:50:08 dist-test-slave-g1xh krb5kdc[3846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304608, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:08.714741 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:08.676932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:46037 (local address 127.2.172.126:37161)
0925 02:50:08.677594 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:08.677604 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:08.679508 (+ 1904us) server_negotiation.cc:407] Connection header received
0925 02:50:08.683988 (+ 4480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:08.683997 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:08.684149 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:08.684321 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:08.687413 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.688963 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.691933 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.692757 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.692883 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:08.704627 (+ 11744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:08.704639 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:08.704646 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:08.704711 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:08.709193 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:08.710534 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:08.710542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:08.710548 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:08.710742 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:08.711581 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:08.711588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:08.711593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:08.712075 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:08.712278 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:08.713746 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0925 02:50:08.714056 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":340,"threads_started":1}
I20260925 02:50:08.717499 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:08.677863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:37161 (local address 127.2.172.66:46037)
0925 02:50:08.679245 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:08.679326 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:08.683384 (+ 4058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:08.684892 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:08.684910 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:08.686017 (+ 1107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:08.687119 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:08.687151 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.689213 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.689219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:08.691701 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:08.691722 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.697179 (+ 5457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.697197 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:08.697726 (+ 529us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:08.699273 (+ 1547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:08.699323 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:08.704121 (+ 4798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:08.709430 (+ 5309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:08.709444 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:08.709480 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:08.710337 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:08.710899 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:08.710906 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:08.710910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:08.711274 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:08.712312 (+ 1038us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:08.712327 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:08.713110 (+ 783us) client_negotiation.cc:769] Sending connection context
0925 02:50:08.713615 (+ 505us) client_negotiation.cc:240] Negotiation successful
0925 02:50:08.714529 (+ 914us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":512,"threads_started":1}
I20260925 02:50:08.747893 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:08.726327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:56341 (local address 127.2.172.126:37161)
0925 02:50:08.726549 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:08.726556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:08.727331 (+ 775us) server_negotiation.cc:407] Connection header received
0925 02:50:08.727535 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:08.727544 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:08.727702 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:08.727841 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:08.729246 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.730561 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.733535 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.734306 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.734426 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:08.738142 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:08.738154 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:08.738162 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:08.738240 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:08.743026 (+ 4786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:08.744371 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:08.744380 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:08.744387 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:08.744587 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:08.745207 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:08.745216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:08.745222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:08.745955 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:08.746134 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:08.747030 (+ 896us) server_negotiation.cc:299] Negotiation successful
0925 02:50:08.747393 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:50:08.749910 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:08.726371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:37161 (local address 127.2.172.66:56341)
0925 02:50:08.727187 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:08.727209 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:08.727380 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:08.727873 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:08.727879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:08.728523 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:08.729041 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:08.729062 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.730791 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.730799 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:08.733350 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:08.733367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:08.734814 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:08.734824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:08.735034 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:08.736469 (+ 1435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:08.736501 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:08.737840 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:08.743267 (+ 5427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:08.743276 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:08.743283 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:08.744167 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:08.744789 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:08.744796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:08.744803 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:08.745026 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:08.746252 (+ 1226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:08.746260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:08.746582 (+ 322us) client_negotiation.cc:769] Sending connection context
0925 02:50:08.748610 (+ 2028us) client_negotiation.cc:240] Negotiation successful
0925 02:50:08.749170 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":462,"threads_started":1}
I20260925 02:50:08.752067 4105 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:34735
I20260925 02:50:08.752231 4241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:34735 every 8 connection(s)
I20260925 02:50:08.753948 4105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:08.756680 4242 heartbeater.cc:347] Connected to a master server at 127.2.172.126:37161
I20260925 02:50:08.756955 4242 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:08.757733 4242 heartbeater.cc:511] Master 127.2.172.126:37161 requested a full tablet report, sending...
I20260925 02:50:08.759469 3891 ts_manager.cc:194] Registered new tserver with Master: 86514a238b3e4b299c385f16229980b6 (127.2.172.66:34735)
I20260925 02:50:08.760586 3891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:46037
I20260925 02:50:08.763674 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 4105
I20260925 02:50:08.763931 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:08.764632 2737 external_mini_cluster.cc:1428] Setting key 060b6ef8eda003a4558f16763c78878c
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:50:08.819661 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:37161
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:50:09.096581 4101 heartbeater.cc:503] Master 127.2.172.126:37161 was elected leader, sending a full tablet report...
W20260925 02:50:09.230263 4246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:09.230697 4246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:09.230822 4246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:09.240460 4246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:09.240633 4246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:09.240729 4246 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:50:09.240818 4246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:50:09.253062 4246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.172.126:37161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:09.254981 4246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:09.257043 4246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:09.274899 4254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:09.274886 4253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:09.275976 4256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:09.281234 4246 server_base.cc:1034] running on GCE node
I20260925 02:50:09.282222 4246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:09.283669 4246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:09.284948 4246 hybrid_clock.cc:648] HybridClock initialized: now 1790304609284864 us; error 51 us; skew 500 ppm
I20260925 02:50:09.285343 4246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:09 dist-test-slave-g1xh krb5kdc[3846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304609, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:09.289625 4246 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:50:09.291761 4246 webserver.cc:460] Webserver started at http://127.2.172.67:46755/ using document root <none> and password file <none>
I20260925 02:50:09.292845 4246 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:09.293032 4246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:09.293474 4246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:09.297673 4246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "a04284922c3c47248e9d97d8b2ad413d"
format_stamp: "Formatted at 2026-09-25 02:50:09 on dist-test-slave-g1xh"
server_key: "890cc51a0cfbc39dd46a9cffe60272e7"
server_key_iv: "dfb87ad7909e838ebae9a170246b402b"
server_key_version: "encryptionkey@0"
I20260925 02:50:09.298713 4246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "a04284922c3c47248e9d97d8b2ad413d"
format_stamp: "Formatted at 2026-09-25 02:50:09 on dist-test-slave-g1xh"
server_key: "890cc51a0cfbc39dd46a9cffe60272e7"
server_key_iv: "dfb87ad7909e838ebae9a170246b402b"
server_key_version: "encryptionkey@0"
I20260925 02:50:09.305945 4246 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260925 02:50:09.310720 4263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:09.312822 4246 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260925 02:50:09.313074 4246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "a04284922c3c47248e9d97d8b2ad413d"
format_stamp: "Formatted at 2026-09-25 02:50:09 on dist-test-slave-g1xh"
server_key: "890cc51a0cfbc39dd46a9cffe60272e7"
server_key_iv: "dfb87ad7909e838ebae9a170246b402b"
server_key_version: "encryptionkey@0"
I20260925 02:50:09.313377 4246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:09.340056 4246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:09.346431 4246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:09.346838 4246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:09.351276 4246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:09.351425 4246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:09.351620 4246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:09.351712 4246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 25 02:50:09 dist-test-slave-g1xh krb5kdc[3846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304609, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:09.383309 4273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.352356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:37099 (local address 127.2.172.126:37161)
0925 02:50:09.352986 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:09.352991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:09.354598 (+ 1607us) server_negotiation.cc:407] Connection header received
0925 02:50:09.357106 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:09.357113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:09.357224 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:09.357399 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:09.361172 (+ 3773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.362346 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.365246 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.366237 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.366404 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.373704 (+ 7300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:09.373716 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:09.373722 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:09.373795 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:09.377794 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:09.379388 (+ 1594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:09.379394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:09.379396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:09.379725 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:09.380551 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:09.380559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:09.380564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:09.381053 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:09.381272 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:09.382507 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0925 02:50:09.382814 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":335,"threads_started":1}
I20260925 02:50:09.384987 4274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.353377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:37161 (local address 127.2.172.67:37099)
0925 02:50:09.354376 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:09.354425 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:09.356734 (+ 2309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:09.357857 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:09.357878 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:09.359209 (+ 1331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:09.360804 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.360859 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.362565 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.362574 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.365033 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.365052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.366790 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.366804 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.367201 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.368525 (+ 1324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:09.368584 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:09.373280 (+ 4696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:09.378041 (+ 4761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:09.378063 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:09.378100 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:09.379182 (+ 1082us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:09.379901 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:09.379914 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:09.379921 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:09.380362 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:09.381386 (+ 1024us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:09.381400 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:09.381978 (+ 578us) client_negotiation.cc:769] Sending connection context
0925 02:50:09.382386 (+ 408us) client_negotiation.cc:240] Negotiation successful
0925 02:50:09.383197 (+ 811us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":431,"threads_started":1}
I20260925 02:50:09.414242 4273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.394612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:47035 (local address 127.2.172.126:37161)
0925 02:50:09.394914 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:09.394926 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:09.395732 (+ 806us) server_negotiation.cc:407] Connection header received
0925 02:50:09.396053 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:09.396058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:09.396167 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:09.396305 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:09.397514 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.398371 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.400302 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.401013 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.401191 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.404386 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:09.404391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:09.404395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:09.404440 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:09.408944 (+ 4504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:09.410377 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:09.410387 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:09.410392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:09.410587 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:09.411081 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:09.411086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:09.411090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:09.411457 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:09.411692 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:09.413367 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0925 02:50:09.413726 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 02:50:09.414952 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.394570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:37161 (local address 127.2.172.67:47035)
0925 02:50:09.395515 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:09.395581 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:09.395864 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:09.396341 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:09.396346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:09.396961 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:09.397315 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.397327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.398599 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.398603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.400100 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.400126 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.401253 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.401260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.401457 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.402493 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:09.402514 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:09.404104 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:09.409153 (+ 5049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:09.409161 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:09.409166 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:09.410103 (+ 937us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:09.410737 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:09.410743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:09.410747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:09.410929 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:09.411649 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:09.411659 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:09.411929 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 02:50:09.412146 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 02:50:09.412692 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":472,"threads_started":1}
I20260925 02:50:09.417330 4246 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:38389
I20260925 02:50:09.417444 4386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:38389 every 8 connection(s)
I20260925 02:50:09.419358 4246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:09.422688 4387 heartbeater.cc:347] Connected to a master server at 127.2.172.126:37161
I20260925 02:50:09.423038 4387 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:09.424017 4387 heartbeater.cc:511] Master 127.2.172.126:37161 requested a full tablet report, sending...
I20260925 02:50:09.426064 3891 ts_manager.cc:194] Registered new tserver with Master: a04284922c3c47248e9d97d8b2ad413d (127.2.172.67:38389)
I20260925 02:50:09.427219 3891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:37099
I20260925 02:50:09.429716 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 4246
I20260925 02:50:09.429973 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:09.430610 2737 external_mini_cluster.cc:1428] Setting key a326ef3026d1e9b7fe40b6d5cc2858cd
I20260925 02:50:09.434943 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:50:09 dist-test-slave-g1xh krb5kdc[3846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:50:09 dist-test-slave-g1xh krb5kdc[3846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:09.500492 4273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.480941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45238 (local address 127.2.172.126:37161)
0925 02:50:09.481271 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:09.481279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:09.481538 (+ 259us) server_negotiation.cc:407] Connection header received
0925 02:50:09.481895 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:09.481901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:09.481993 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:09.482133 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:09.484115 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.485017 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.486658 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.487204 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.487389 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.493212 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:09.493220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:09.493224 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:09.493295 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:09.496910 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:09.497884 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:09.497899 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:09.497901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:09.498077 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:09.498581 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:09.498595 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:09.498598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:09.498978 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:09.499306 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:09.499773 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 02:50:09.500056 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 02:50:09.507180 3891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:09.516723 3891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:09.561432 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.536129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43533 (local address 127.0.0.1:51344)
0925 02:50:09.544720 (+ 8591us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:09.544829 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:09.546514 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:09.548282 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:09.548297 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:09.548351 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:09.549360 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.549392 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.552629 (+ 3237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.552639 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.556339 (+ 3700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.556377 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.560045 (+ 3668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.560051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.560400 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.560480 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 02:50:09.560668 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 02:50:09.560945 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8085,"thread_start_us":9245,"threads_started":1}
I20260925 02:50:09.562598 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.536800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.2.172.65:43533)
0925 02:50:09.537773 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:09.537788 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:09.545075 (+ 7287us) server_negotiation.cc:407] Connection header received
0925 02:50:09.546839 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:09.546854 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:09.547387 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:09.547733 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:09.549773 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.552262 (+ 2489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.557444 (+ 5182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.559352 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.559500 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.559851 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:09.561949 (+ 2098us) server_negotiation.cc:299] Negotiation successful
0925 02:50:09.562186 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"spinlock_wait_cycles":44928,"thread_start_us":547,"threads_started":1}
I20260925 02:50:09.568135 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.536194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:38389 (local address 127.0.0.1:43030)
0925 02:50:09.545806 (+ 9612us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:09.545833 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:09.546149 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:09.559074 (+ 12925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:09.559081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:09.559129 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:09.559732 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.559759 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.562381 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.562387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.564756 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.564773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.567059 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.567065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.567276 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.567326 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 02:50:09.567408 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:50:09.567610 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9399,"mutex_wait_us":1,"thread_start_us":28395,"threads_started":1}
I20260925 02:50:09.568666 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.536900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43030 (local address 127.2.172.67:38389)
0925 02:50:09.557921 (+ 21021us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:09.557931 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:09.557978 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:50:09.558145 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:09.558156 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:09.558750 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:09.560004 (+ 1254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:09.560446 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.562200 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.565019 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.566733 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.567681 (+ 948us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.567982 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:09.568086 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 02:50:09.568297 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20785,"thread_start_us":20554,"threads_started":1}
I20260925 02:50:09.569326 4031 tablet_service.cc:1467] Processing CreateTablet for tablet 00fda2dd4aad419984f4e1b322385f23 (DEFAULT_TABLE table=test-table [id=7bb050a9f62346acb61c23c0e05f4d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:09.571506 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.536653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:34735 (local address 127.0.0.1:57652)
0925 02:50:09.563417 (+ 26764us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:09.563441 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:09.563619 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:09.564884 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:09.564892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:09.564934 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:09.565132 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.565143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.567053 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.567059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.569024 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.569038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.570720 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.570726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.570911 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.570946 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 02:50:09.571032 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:50:09.571193 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26561,"thread_start_us":7337,"threads_started":1}
I20260925 02:50:09.571674 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.537083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.2.172.66:34735)
0925 02:50:09.557203 (+ 20120us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:09.557216 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:09.563806 (+ 6590us) server_negotiation.cc:407] Connection header received
0925 02:50:09.563961 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:09.563971 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:09.564347 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:09.564611 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:09.565325 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.566845 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.569201 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.570536 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.570709 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.570985 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:09.571138 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 02:50:09.571330 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19897,"thread_start_us":19705,"threads_started":1}
I20260925 02:50:09.572628 4031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fda2dd4aad419984f4e1b322385f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:09.573666 4316 tablet_service.cc:1467] Processing CreateTablet for tablet 00fda2dd4aad419984f4e1b322385f23 (DEFAULT_TABLE table=test-table [id=7bb050a9f62346acb61c23c0e05f4d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:09.575701 4316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fda2dd4aad419984f4e1b322385f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:09.575887 4172 tablet_service.cc:1467] Processing CreateTablet for tablet 00fda2dd4aad419984f4e1b322385f23 (DEFAULT_TABLE table=test-table [id=7bb050a9f62346acb61c23c0e05f4d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:09.577667 4172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fda2dd4aad419984f4e1b322385f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:09.592000 4406 tablet_bootstrap.cc:492] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4: Bootstrap starting.
I20260925 02:50:09.593191 4407 tablet_bootstrap.cc:492] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6: Bootstrap starting.
I20260925 02:50:09.595130 4408 tablet_bootstrap.cc:492] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d: Bootstrap starting.
I20260925 02:50:09.598001 4407 tablet_bootstrap.cc:654] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:09.598054 4406 tablet_bootstrap.cc:654] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:09.599259 4408 tablet_bootstrap.cc:654] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:09.599274 4407 log.cc:824] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:09.599419 4406 log.cc:824] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:09.600648 4408 log.cc:824] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:09.602948 4406 tablet_bootstrap.cc:492] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4: No bootstrap required, opened a new log
I20260925 02:50:09.603049 4407 tablet_bootstrap.cc:492] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6: No bootstrap required, opened a new log
I20260925 02:50:09.603312 4406 ts_tablet_manager.cc:1397] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260925 02:50:09.603358 4407 ts_tablet_manager.cc:1397] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 02:50:09.603884 4408 tablet_bootstrap.cc:492] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d: No bootstrap required, opened a new log
I20260925 02:50:09.604159 4408 ts_tablet_manager.cc:1397] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20260925 02:50:09.612736 4406 raft_consensus.cc:348] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } }
I20260925 02:50:09.613459 4406 raft_consensus.cc:374] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:09.613591 4406 raft_consensus.cc:729] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac5091b06704f17b5fe94b423a033b4, State: Initialized, Role: FOLLOWER
I20260925 02:50:09.614487 4406 consensus_queue.cc:260] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } }
I20260925 02:50:09.615518 4407 raft_consensus.cc:348] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } }
I20260925 02:50:09.616149 4407 raft_consensus.cc:374] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:09.616297 4407 raft_consensus.cc:729] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86514a238b3e4b299c385f16229980b6, State: Initialized, Role: FOLLOWER
I20260925 02:50:09.615998 4408 raft_consensus.cc:348] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } }
I20260925 02:50:09.616545 4408 raft_consensus.cc:374] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:09.616696 4408 raft_consensus.cc:729] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04284922c3c47248e9d97d8b2ad413d, State: Initialized, Role: FOLLOWER
I20260925 02:50:09.616894 4406 ts_tablet_manager.cc:1428] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 02:50:09.617197 4407 consensus_queue.cc:260] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } }
I20260925 02:50:09.617492 4408 consensus_queue.cc:260] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } }
I20260925 02:50:09.624512 4242 heartbeater.cc:503] Master 127.2.172.126:37161 was elected leader, sending a full tablet report...
I20260925 02:50:09.625727 4408 ts_tablet_manager.cc:1428] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d: Time spent starting tablet: real 0.021s user 0.017s sys 0.004s
I20260925 02:50:09.625747 4407 ts_tablet_manager.cc:1428] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6: Time spent starting tablet: real 0.022s user 0.010s sys 0.010s
I20260925 02:50:09.626302 4387 heartbeater.cc:503] Master 127.2.172.126:37161 was elected leader, sending a full tablet report...
W20260925 02:50:09.672468 4388 tablet.cc:2369] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:09.684434 4412 raft_consensus.cc:482] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:09.685010 4412 raft_consensus.cc:504] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } }
I20260925 02:50:09.687857 4412 leader_election.cc:288] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04284922c3c47248e9d97d8b2ad413d (127.2.172.67:38389), 86514a238b3e4b299c385f16229980b6 (127.2.172.66:34735)
I20260925 02:50:09.697616 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.688332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:60877 (local address 127.2.172.67:38389)
0925 02:50:09.688758 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:09.688768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:09.690441 (+ 1673us) server_negotiation.cc:407] Connection header received
0925 02:50:09.690673 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:09.690679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:09.690757 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:09.690861 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:09.692018 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.692969 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.695364 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.696390 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.696521 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.696591 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:09.697137 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 02:50:09.697293 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 02:50:09.697867 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.688643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:38389 (local address 127.2.172.65:60877)
0925 02:50:09.690291 (+ 1648us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:09.690318 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:09.690503 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:09.690877 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:09.690881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:09.690914 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:09.691682 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.691695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.693117 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.693125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.695142 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.695161 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.696807 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.696812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.696945 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.696974 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 02:50:09.697098 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 02:50:09.697279 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"mutex_wait_us":68,"thread_start_us":1132,"threads_started":1}
I20260925 02:50:09.697932 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.688714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:48827 (local address 127.2.172.66:34735)
0925 02:50:09.688969 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:09.688974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:09.689979 (+ 1005us) server_negotiation.cc:407] Connection header received
0925 02:50:09.690272 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:09.690276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:09.690351 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:09.690457 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:09.691093 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.692210 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.695377 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.696616 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.696856 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.696996 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:09.697412 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 02:50:09.697571 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260925 02:50:09.698356 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.688656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:34735 (local address 127.2.172.65:48827)
0925 02:50:09.689817 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:09.689836 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:09.690089 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:09.690483 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:09.690488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:09.690528 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:09.690925 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.690935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.692395 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.692399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.695145 (+ 2746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:09.695160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.696995 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.697004 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:09.697222 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.697249 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:50:09.697375 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 02:50:09.697530 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":28,"thread_start_us":638,"threads_started":1}
I20260925 02:50:09.699879 4340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fda2dd4aad419984f4e1b322385f23" candidate_uuid: "fac5091b06704f17b5fe94b423a033b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04284922c3c47248e9d97d8b2ad413d" is_pre_election: true
I20260925 02:50:09.699867 4192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fda2dd4aad419984f4e1b322385f23" candidate_uuid: "fac5091b06704f17b5fe94b423a033b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86514a238b3e4b299c385f16229980b6" is_pre_election: true
I20260925 02:50:09.700891 4340 raft_consensus.cc:2454] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac5091b06704f17b5fe94b423a033b4 in term 0.
I20260925 02:50:09.700896 4192 raft_consensus.cc:2454] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac5091b06704f17b5fe94b423a033b4 in term 0.
I20260925 02:50:09.702097 3980 leader_election.cc:302] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04284922c3c47248e9d97d8b2ad413d, fac5091b06704f17b5fe94b423a033b4; no voters:
I20260925 02:50:09.702739 4412 raft_consensus.cc:2781] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:09.703056 4412 raft_consensus.cc:482] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:09.703246 4412 raft_consensus.cc:3037] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:09.706017 4412 raft_consensus.cc:504] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } }
I20260925 02:50:09.706878 4412 leader_election.cc:288] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [CANDIDATE]: Term 1 election: Requested vote from peers a04284922c3c47248e9d97d8b2ad413d (127.2.172.67:38389), 86514a238b3e4b299c385f16229980b6 (127.2.172.66:34735)
I20260925 02:50:09.707616 4340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fda2dd4aad419984f4e1b322385f23" candidate_uuid: "fac5091b06704f17b5fe94b423a033b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04284922c3c47248e9d97d8b2ad413d"
I20260925 02:50:09.707643 4192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fda2dd4aad419984f4e1b322385f23" candidate_uuid: "fac5091b06704f17b5fe94b423a033b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86514a238b3e4b299c385f16229980b6"
I20260925 02:50:09.707953 4192 raft_consensus.cc:3037] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:09.707947 4340 raft_consensus.cc:3037] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:09.710593 4192 raft_consensus.cc:2454] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac5091b06704f17b5fe94b423a033b4 in term 1.
I20260925 02:50:09.710600 4340 raft_consensus.cc:2454] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac5091b06704f17b5fe94b423a033b4 in term 1.
I20260925 02:50:09.711189 3980 leader_election.cc:302] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04284922c3c47248e9d97d8b2ad413d, fac5091b06704f17b5fe94b423a033b4; no voters:
I20260925 02:50:09.711601 4412 raft_consensus.cc:2781] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:09.712185 4412 raft_consensus.cc:686] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 1 LEADER]: Becoming Leader. State: Replica: fac5091b06704f17b5fe94b423a033b4, State: Running, Role: LEADER
I20260925 02:50:09.712913 4412 consensus_queue.cc:237] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } }
I20260925 02:50:09.719993 3892 catalog_manager.cc:5495] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fac5091b06704f17b5fe94b423a033b4 (127.2.172.65). New cstate: current_term: 1 leader_uuid: "fac5091b06704f17b5fe94b423a033b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } health_report { overall_health: UNKNOWN } } }
W20260925 02:50:09.756681 4243 tablet.cc:2369] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:09.757718 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:09.749834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.2.172.65:43533)
0925 02:50:09.750260 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:09.750268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:09.750466 (+ 198us) server_negotiation.cc:407] Connection header received
0925 02:50:09.750993 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:09.750998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:09.751120 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:09.751307 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:09.751871 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.752727 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.754335 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:09.754878 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:09.755039 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:09.755880 (+ 841us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:09.756615 (+ 735us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:09.756831 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:09.757358 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 02:50:09.757472 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 02:50:09.769697 4340 raft_consensus.cc:1261] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d [term 1 FOLLOWER]: Refusing update from remote peer fac5091b06704f17b5fe94b423a033b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:09.769716 4192 raft_consensus.cc:1261] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6 [term 1 FOLLOWER]: Refusing update from remote peer fac5091b06704f17b5fe94b423a033b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:09.771173 4417 consensus_queue.cc:1035] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:09.773358 4417 consensus_queue.cc:1035] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:09.790532 4421 mvcc.cc:205] Tried to move back new op lower bound from 7333087681603248128 to 7333087681390977024. Current Snapshot: MvccSnapshot[applied={T|T < 7333087681603248128}]
I20260925 02:50:09.815101 3892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:09.815593 3892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:09.821291 3892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7: Sending DeleteTablet for 3 replicas of tablet 00fda2dd4aad419984f4e1b322385f23
I20260925 02:50:09.823122 4172 tablet_service.cc:1514] Processing DeleteTablet for tablet 00fda2dd4aad419984f4e1b322385f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:09 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:57652
I20260925 02:50:09.824584 4430 tablet_replica.cc:330] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6: stopping tablet replica
I20260925 02:50:09.823129 4316 tablet_service.cc:1514] Processing DeleteTablet for tablet 00fda2dd4aad419984f4e1b322385f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:09 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:43030
I20260925 02:50:09.823127 4031 tablet_service.cc:1514] Processing DeleteTablet for tablet 00fda2dd4aad419984f4e1b322385f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:09 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:51344
I20260925 02:50:09.825515 4430 raft_consensus.cc:2229] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:09.826244 4430 raft_consensus.cc:2258] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:09.828771 3891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:50:09.829460 4430 ts_tablet_manager.cc:1905] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 02:50:09.829828 3891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:09.834146 4432 tablet_replica.cc:330] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4: stopping tablet replica
I20260925 02:50:09.837836 4432 raft_consensus.cc:2229] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:50:09.839495 4432 raft_consensus.cc:2258] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:09.839774 4431 tablet_replica.cc:330] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d: stopping tablet replica
I20260925 02:50:09.840581 4431 raft_consensus.cc:2229] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:09.841084 4430 ts_tablet_manager.cc:1918] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:09.841430 4430 log.cc:1196] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/00fda2dd4aad419984f4e1b322385f23
I20260925 02:50:09.842283 4430 ts_tablet_manager.cc:1939] T 00fda2dd4aad419984f4e1b322385f23 P 86514a238b3e4b299c385f16229980b6: Deleting consensus metadata
I20260925 02:50:09.842531 4432 ts_tablet_manager.cc:1905] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:09.842921 4172 tablet_service.cc:1467] Processing CreateTablet for tablet 076b80e0264441fa9026c5296faa8d89 (DEFAULT_TABLE table=test-table [id=eabddb86d25c4d398999dd1c3e768202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:09.843839 4172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076b80e0264441fa9026c5296faa8d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:09.844542 4031 tablet_service.cc:1467] Processing CreateTablet for tablet 076b80e0264441fa9026c5296faa8d89 (DEFAULT_TABLE table=test-table [id=eabddb86d25c4d398999dd1c3e768202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:09.845121 4031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076b80e0264441fa9026c5296faa8d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:09.843178 4316 tablet_service.cc:1467] Processing CreateTablet for tablet 076b80e0264441fa9026c5296faa8d89 (DEFAULT_TABLE table=test-table [id=eabddb86d25c4d398999dd1c3e768202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:09.847011 4316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076b80e0264441fa9026c5296faa8d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:09.852824 4407 tablet_bootstrap.cc:492] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6: Bootstrap starting.
I20260925 02:50:09.852787 3876 catalog_manager.cc:4841] TS 86514a238b3e4b299c385f16229980b6 (127.2.172.66:34735): tablet 00fda2dd4aad419984f4e1b322385f23 (table test-table [id=7bb050a9f62346acb61c23c0e05f4d41]) successfully deleted
I20260925 02:50:09.853928 4431 raft_consensus.cc:2258] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:09.854067 4408 tablet_bootstrap.cc:492] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d: Bootstrap starting.
I20260925 02:50:09.855010 4407 tablet_bootstrap.cc:654] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:09.856191 4408 tablet_bootstrap.cc:654] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:09.861334 4406 tablet_bootstrap.cc:492] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4: Bootstrap starting.
I20260925 02:50:09.863024 4432 ts_tablet_manager.cc:1918] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:09.863246 4432 log.cc:1196] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/00fda2dd4aad419984f4e1b322385f23
I20260925 02:50:09.863600 4406 tablet_bootstrap.cc:654] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:09.864187 4431 ts_tablet_manager.cc:1905] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:09.868224 4432 ts_tablet_manager.cc:1939] T 00fda2dd4aad419984f4e1b322385f23 P fac5091b06704f17b5fe94b423a033b4: Deleting consensus metadata
I20260925 02:50:09.868453 4408 tablet_bootstrap.cc:492] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d: No bootstrap required, opened a new log
I20260925 02:50:09.869127 4408 ts_tablet_manager.cc:1397] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260925 02:50:09.870265 4408 raft_consensus.cc:348] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:09.870548 4408 raft_consensus.cc:374] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:09.870648 4408 raft_consensus.cc:729] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04284922c3c47248e9d97d8b2ad413d, State: Initialized, Role: FOLLOWER
I20260925 02:50:09.871016 4408 consensus_queue.cc:260] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:09.871802 4408 ts_tablet_manager.cc:1428] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:09.873243 3877 catalog_manager.cc:4841] TS fac5091b06704f17b5fe94b423a033b4 (127.2.172.65:43533): tablet 00fda2dd4aad419984f4e1b322385f23 (table test-table [id=7bb050a9f62346acb61c23c0e05f4d41]) successfully deleted
I20260925 02:50:09.874989 4406 tablet_bootstrap.cc:492] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4: No bootstrap required, opened a new log
I20260925 02:50:09.876117 4406 ts_tablet_manager.cc:1397] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20260925 02:50:09.876389 4407 tablet_bootstrap.cc:492] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6: No bootstrap required, opened a new log
I20260925 02:50:09.876585 4407 ts_tablet_manager.cc:1397] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.004s
I20260925 02:50:09.877732 4407 raft_consensus.cc:348] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:09.878051 4407 raft_consensus.cc:374] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:09.878170 4407 raft_consensus.cc:729] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86514a238b3e4b299c385f16229980b6, State: Initialized, Role: FOLLOWER
I20260925 02:50:09.878342 4431 ts_tablet_manager.cc:1918] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:09.878599 4431 log.cc:1196] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/00fda2dd4aad419984f4e1b322385f23
I20260925 02:50:09.878520 4407 consensus_queue.cc:260] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:09.879285 4431 ts_tablet_manager.cc:1939] T 00fda2dd4aad419984f4e1b322385f23 P a04284922c3c47248e9d97d8b2ad413d: Deleting consensus metadata
I20260925 02:50:09.882222 4407 ts_tablet_manager.cc:1428] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:50:09.882177 3875 catalog_manager.cc:4841] TS a04284922c3c47248e9d97d8b2ad413d (127.2.172.67:38389): tablet 00fda2dd4aad419984f4e1b322385f23 (table test-table [id=7bb050a9f62346acb61c23c0e05f4d41]) successfully deleted
I20260925 02:50:09.883121 4406 raft_consensus.cc:348] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:09.883399 4406 raft_consensus.cc:374] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:09.883615 4406 raft_consensus.cc:729] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac5091b06704f17b5fe94b423a033b4, State: Initialized, Role: FOLLOWER
I20260925 02:50:09.884200 4406 consensus_queue.cc:260] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:09.885068 4406 ts_tablet_manager.cc:1428] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20260925 02:50:10.004386 4425 raft_consensus.cc:482] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:10.004819 4425 raft_consensus.cc:504] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:10.005959 4425 leader_election.cc:288] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86514a238b3e4b299c385f16229980b6 (127.2.172.66:34735), a04284922c3c47248e9d97d8b2ad413d (127.2.172.67:38389)
I20260925 02:50:10.006768 4340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076b80e0264441fa9026c5296faa8d89" candidate_uuid: "fac5091b06704f17b5fe94b423a033b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04284922c3c47248e9d97d8b2ad413d" is_pre_election: true
I20260925 02:50:10.006798 4192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076b80e0264441fa9026c5296faa8d89" candidate_uuid: "fac5091b06704f17b5fe94b423a033b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86514a238b3e4b299c385f16229980b6" is_pre_election: true
I20260925 02:50:10.007117 4340 raft_consensus.cc:2454] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac5091b06704f17b5fe94b423a033b4 in term 0.
I20260925 02:50:10.007112 4192 raft_consensus.cc:2454] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac5091b06704f17b5fe94b423a033b4 in term 0.
I20260925 02:50:10.007704 3981 leader_election.cc:302] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86514a238b3e4b299c385f16229980b6, fac5091b06704f17b5fe94b423a033b4; no voters:
I20260925 02:50:10.008051 4425 raft_consensus.cc:2781] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:10.008188 4425 raft_consensus.cc:482] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:10.008289 4425 raft_consensus.cc:3037] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:10.012080 4425 raft_consensus.cc:504] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:10.012784 4425 leader_election.cc:288] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [CANDIDATE]: Term 1 election: Requested vote from peers 86514a238b3e4b299c385f16229980b6 (127.2.172.66:34735), a04284922c3c47248e9d97d8b2ad413d (127.2.172.67:38389)
I20260925 02:50:10.013423 4340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076b80e0264441fa9026c5296faa8d89" candidate_uuid: "fac5091b06704f17b5fe94b423a033b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04284922c3c47248e9d97d8b2ad413d"
I20260925 02:50:10.013420 4192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076b80e0264441fa9026c5296faa8d89" candidate_uuid: "fac5091b06704f17b5fe94b423a033b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86514a238b3e4b299c385f16229980b6"
I20260925 02:50:10.013670 4340 raft_consensus.cc:3037] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:10.013712 4192 raft_consensus.cc:3037] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:10.016835 4192 raft_consensus.cc:2454] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac5091b06704f17b5fe94b423a033b4 in term 1.
I20260925 02:50:10.017077 4340 raft_consensus.cc:2454] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac5091b06704f17b5fe94b423a033b4 in term 1.
I20260925 02:50:10.017351 3981 leader_election.cc:302] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86514a238b3e4b299c385f16229980b6, fac5091b06704f17b5fe94b423a033b4; no voters:
I20260925 02:50:10.017724 4425 raft_consensus.cc:2781] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:10.017889 4425 raft_consensus.cc:686] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 1 LEADER]: Becoming Leader. State: Replica: fac5091b06704f17b5fe94b423a033b4, State: Running, Role: LEADER
I20260925 02:50:10.018246 4425 consensus_queue.cc:237] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:10.021571 3890 catalog_manager.cc:5495] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fac5091b06704f17b5fe94b423a033b4 (127.2.172.65). New cstate: current_term: 1 leader_uuid: "fac5091b06704f17b5fe94b423a033b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } health_report { overall_health: UNKNOWN } } }
W20260925 02:50:10.086374 4102 tablet.cc:2369] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:50:10 dist-test-slave-g1xh krb5kdc[3846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304607, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:10.122025 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:10.090176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:37161 (local address 127.0.0.1:45250)
0925 02:50:10.104158 (+ 13982us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:10.104194 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:10.104412 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:10.104922 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:10.104927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:10.105900 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:10.106233 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:10.106247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.108035 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.108041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:10.109580 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:10.109597 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.110715 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.110722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:10.110905 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:10.111817 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:10.111842 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:10.115175 (+ 3333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:10.119079 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:10.119093 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:10.119100 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:10.119508 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:10.119962 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:10.119968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:10.119972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:10.120120 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:10.120765 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:10.120774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:10.120930 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 02:50:10.121131 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 02:50:10.121451 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13714,"thread_start_us":13567,"threads_started":1}
I20260925 02:50:10.122031 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:10.090162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45250 (local address 127.2.172.126:37161)
0925 02:50:10.104508 (+ 14346us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:10.104516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:10.104543 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:50:10.104619 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:10.104626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:10.104717 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:10.104855 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:10.106464 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.107860 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.109768 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.110522 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.110655 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:10.115464 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:10.115471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:10.115477 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:10.115524 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:10.118865 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:10.119706 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:10.119713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:10.119717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:10.119850 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:10.120237 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:10.120244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:10.120247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:10.120636 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:10.120795 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:10.121241 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 02:50:10.121650 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":14201,"thread_start_us":14051,"threads_started":1}
I20260925 02:50:10.129179 3890 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:45250:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:50:10.130564 3890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:10.138080 4172 tablet_service.cc:1467] Processing CreateTablet for tablet 2f56c3e7105b4a59853fbf120b7a4059 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e7692751ecbc49f4a14f88ede72145e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:50:10.138175 4316 tablet_service.cc:1467] Processing CreateTablet for tablet 2f56c3e7105b4a59853fbf120b7a4059 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e7692751ecbc49f4a14f88ede72145e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:50:10.138103 4031 tablet_service.cc:1467] Processing CreateTablet for tablet 2f56c3e7105b4a59853fbf120b7a4059 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e7692751ecbc49f4a14f88ede72145e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:50:10.138757 4316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f56c3e7105b4a59853fbf120b7a4059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:10.138769 4172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f56c3e7105b4a59853fbf120b7a4059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:10.138788 4031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f56c3e7105b4a59853fbf120b7a4059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:10.146422 4407 tablet_bootstrap.cc:492] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6: Bootstrap starting.
I20260925 02:50:10.147930 4407 tablet_bootstrap.cc:654] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:10.148728 4406 tablet_bootstrap.cc:492] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: Bootstrap starting.
I20260925 02:50:10.150671 4407 tablet_bootstrap.cc:492] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6: No bootstrap required, opened a new log
I20260925 02:50:10.150858 4406 tablet_bootstrap.cc:654] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:10.150848 4407 ts_tablet_manager.cc:1397] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 02:50:10.151739 4407 raft_consensus.cc:348] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:10.152006 4407 raft_consensus.cc:374] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:10.152134 4407 raft_consensus.cc:729] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86514a238b3e4b299c385f16229980b6, State: Initialized, Role: FOLLOWER
I20260925 02:50:10.152426 4407 consensus_queue.cc:260] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:10.153071 4413 tablet_replica.cc:439] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } }
I20260925 02:50:10.153506 4413 tablet_replica.cc:442] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:10.154237 4408 tablet_bootstrap.cc:492] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d: Bootstrap starting.
I20260925 02:50:10.154968 4406 tablet_bootstrap.cc:492] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: No bootstrap required, opened a new log
I20260925 02:50:10.155150 4406 ts_tablet_manager.cc:1397] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 02:50:10.155853 4407 ts_tablet_manager.cc:1428] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:50:10.156028 4406 raft_consensus.cc:348] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:10.156282 4406 raft_consensus.cc:374] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:10.156361 4406 raft_consensus.cc:729] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac5091b06704f17b5fe94b423a033b4, State: Initialized, Role: FOLLOWER
I20260925 02:50:10.156416 4408 tablet_bootstrap.cc:654] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:10.156934 4406 consensus_queue.cc:260] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:10.157652 4406 ts_tablet_manager.cc:1428] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:50:10.157498 4425 tablet_replica.cc:439] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } }
I20260925 02:50:10.157853 4425 tablet_replica.cc:442] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:10.161899 4408 tablet_bootstrap.cc:492] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d: No bootstrap required, opened a new log
I20260925 02:50:10.162117 4408 ts_tablet_manager.cc:1397] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260925 02:50:10.162950 4408 raft_consensus.cc:348] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:10.163179 4408 raft_consensus.cc:374] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:10.163283 4408 raft_consensus.cc:729] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04284922c3c47248e9d97d8b2ad413d, State: Initialized, Role: FOLLOWER
I20260925 02:50:10.163537 4408 consensus_queue.cc:260] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:10.164170 4408 ts_tablet_manager.cc:1428] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:50:10.164060 4414 tablet_replica.cc:439] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } }
I20260925 02:50:10.164392 4414 tablet_replica.cc:442] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:10.415768 4425 consensus_queue.cc:1035] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:10.422168 4425 consensus_queue.cc:1035] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:10.497994 4425 raft_consensus.cc:482] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:10.498260 4425 raft_consensus.cc:504] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:10.498911 4425 leader_election.cc:288] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86514a238b3e4b299c385f16229980b6 (127.2.172.66:34735), a04284922c3c47248e9d97d8b2ad413d (127.2.172.67:38389)
I20260925 02:50:10.499735 4340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f56c3e7105b4a59853fbf120b7a4059" candidate_uuid: "fac5091b06704f17b5fe94b423a033b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04284922c3c47248e9d97d8b2ad413d" is_pre_election: true
I20260925 02:50:10.499955 4192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f56c3e7105b4a59853fbf120b7a4059" candidate_uuid: "fac5091b06704f17b5fe94b423a033b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86514a238b3e4b299c385f16229980b6" is_pre_election: true
I20260925 02:50:10.500061 4340 raft_consensus.cc:2454] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac5091b06704f17b5fe94b423a033b4 in term 0.
I20260925 02:50:10.500212 4192 raft_consensus.cc:2454] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac5091b06704f17b5fe94b423a033b4 in term 0.
I20260925 02:50:10.500552 3980 leader_election.cc:302] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04284922c3c47248e9d97d8b2ad413d, fac5091b06704f17b5fe94b423a033b4; no voters:
I20260925 02:50:10.500885 4425 raft_consensus.cc:2781] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:10.500990 4425 raft_consensus.cc:482] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:10.501111 4425 raft_consensus.cc:3037] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:10.503109 4425 raft_consensus.cc:504] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:10.503722 4425 leader_election.cc:288] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [CANDIDATE]: Term 1 election: Requested vote from peers 86514a238b3e4b299c385f16229980b6 (127.2.172.66:34735), a04284922c3c47248e9d97d8b2ad413d (127.2.172.67:38389)
I20260925 02:50:10.504187 4192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f56c3e7105b4a59853fbf120b7a4059" candidate_uuid: "fac5091b06704f17b5fe94b423a033b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86514a238b3e4b299c385f16229980b6"
I20260925 02:50:10.504231 4340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f56c3e7105b4a59853fbf120b7a4059" candidate_uuid: "fac5091b06704f17b5fe94b423a033b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04284922c3c47248e9d97d8b2ad413d"
I20260925 02:50:10.504443 4192 raft_consensus.cc:3037] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:10.504462 4340 raft_consensus.cc:3037] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:10.506429 4340 raft_consensus.cc:2454] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac5091b06704f17b5fe94b423a033b4 in term 1.
I20260925 02:50:10.506652 4192 raft_consensus.cc:2454] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac5091b06704f17b5fe94b423a033b4 in term 1.
I20260925 02:50:10.506860 3980 leader_election.cc:302] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04284922c3c47248e9d97d8b2ad413d, fac5091b06704f17b5fe94b423a033b4; no voters:
I20260925 02:50:10.507180 4425 raft_consensus.cc:2781] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:10.507421 4425 raft_consensus.cc:686] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [term 1 LEADER]: Becoming Leader. State: Replica: fac5091b06704f17b5fe94b423a033b4, State: Running, Role: LEADER
I20260925 02:50:10.507725 4425 consensus_queue.cc:237] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } }
I20260925 02:50:10.509646 4417 tablet_replica.cc:439] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: TxnStatusTablet state changed. Reason: New leader fac5091b06704f17b5fe94b423a033b4. Latest consensus state: current_term: 1 leader_uuid: "fac5091b06704f17b5fe94b423a033b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } }
I20260925 02:50:10.509902 4417 tablet_replica.cc:442] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:50:10.511247 3891 catalog_manager.cc:5495] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fac5091b06704f17b5fe94b423a033b4 (127.2.172.65). New cstate: current_term: 1 leader_uuid: "fac5091b06704f17b5fe94b423a033b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } health_report { overall_health: UNKNOWN } } }
I20260925 02:50:10.514401 4454 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:50:10.514699 4454 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:50:10.544138 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:10.537332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.2.172.65:43533)
0925 02:50:10.538118 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:10.538123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:10.538140 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:50:10.538204 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:10.538208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:10.538278 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:10.538366 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:10.538859 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.539949 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.541948 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.542380 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.542538 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:10.543177 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:10.543306 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:10.543449 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:10.543684 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 02:50:10.543764 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":480,"threads_started":1}
I20260925 02:50:10.544129 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:10.537322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43533 (local address 127.0.0.1:51354)
0925 02:50:10.537704 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:10.537727 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:10.537889 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:10.538436 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:10.538442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:10.538456 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:50:10.538672 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:10.538683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.540102 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.540110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:10.541778 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:10.541796 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.542599 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.542606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:10.542822 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:10.542913 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:50:10.543424 (+ 511us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:50:10.543479 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 02:50:10.543615 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 02:50:10.543784 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 02:50:10.548832 4192 raft_consensus.cc:1261] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6 [term 1 FOLLOWER]: Refusing update from remote peer fac5091b06704f17b5fe94b423a033b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:10.548827 4340 raft_consensus.cc:1261] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d [term 1 FOLLOWER]: Refusing update from remote peer fac5091b06704f17b5fe94b423a033b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:10.549500 4417 consensus_queue.cc:1035] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:10.549820 4425 consensus_queue.cc:1035] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:10.553712 4414 tablet_replica.cc:439] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d: TxnStatusTablet state changed. Reason: New leader fac5091b06704f17b5fe94b423a033b4. Latest consensus state: current_term: 1 leader_uuid: "fac5091b06704f17b5fe94b423a033b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } }
I20260925 02:50:10.554059 4414 tablet_replica.cc:442] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:10.556511 4413 tablet_replica.cc:439] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6: TxnStatusTablet state changed. Reason: New leader fac5091b06704f17b5fe94b423a033b4. Latest consensus state: current_term: 1 leader_uuid: "fac5091b06704f17b5fe94b423a033b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } }
I20260925 02:50:10.556905 4413 tablet_replica.cc:442] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:10.568229 4414 tablet_replica.cc:439] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fac5091b06704f17b5fe94b423a033b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } }
I20260925 02:50:10.568544 4414 tablet_replica.cc:442] T 2f56c3e7105b4a59853fbf120b7a4059 P a04284922c3c47248e9d97d8b2ad413d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:10.568352 4413 tablet_replica.cc:439] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fac5091b06704f17b5fe94b423a033b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } }
I20260925 02:50:10.568698 4413 tablet_replica.cc:442] T 2f56c3e7105b4a59853fbf120b7a4059 P 86514a238b3e4b299c385f16229980b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:10.576400 4425 tablet_replica.cc:439] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fac5091b06704f17b5fe94b423a033b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } }
I20260925 02:50:10.576815 4425 tablet_replica.cc:442] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:50:10.578186 4417 tablet_replica.cc:439] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fac5091b06704f17b5fe94b423a033b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac5091b06704f17b5fe94b423a033b4" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43533 } } peers { permanent_uuid: "86514a238b3e4b299c385f16229980b6" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 34735 } } peers { permanent_uuid: "a04284922c3c47248e9d97d8b2ad413d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38389 } } }
I20260925 02:50:10.578509 4417 tablet_replica.cc:442] T 2f56c3e7105b4a59853fbf120b7a4059 P fac5091b06704f17b5fe94b423a033b4: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:50:10.587250 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:10.580517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:41605 (local address 127.2.172.65:43533)
0925 02:50:10.580730 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:10.580736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:10.581443 (+ 707us) server_negotiation.cc:407] Connection header received
0925 02:50:10.581599 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:10.581605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:10.581697 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:10.581795 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:10.582349 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.583591 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.585101 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.585789 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.585921 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:10.586431 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:10.586582 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:10.586692 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:10.586856 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 02:50:10.586933 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:50:10.587320 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:10.580537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43533 (local address 127.2.172.65:41605)
0925 02:50:10.581342 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:10.581358 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:10.581480 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:10.581816 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:10.581819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:10.581828 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:50:10.582193 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:10.582201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.583734 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.583738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:10.584928 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:10.584940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.586001 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.586006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:10.586183 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:10.586252 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:50:10.586722 (+ 470us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:50:10.586732 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 02:50:10.586802 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:50:10.586928 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":435,"threads_started":1}
I20260925 02:50:10.661927 3892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:10.662175 3892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:10.665525 3892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dafc9a5e35e4bbb86b0dc74e782eef7: Sending DeleteTablet for 3 replicas of tablet 076b80e0264441fa9026c5296faa8d89
I20260925 02:50:10.666481 4172 tablet_service.cc:1514] Processing DeleteTablet for tablet 076b80e0264441fa9026c5296faa8d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:10 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:57652
I20260925 02:50:10.667042 4316 tablet_service.cc:1514] Processing DeleteTablet for tablet 076b80e0264441fa9026c5296faa8d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:10 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:43030
I20260925 02:50:10.669411 4031 tablet_service.cc:1514] Processing DeleteTablet for tablet 076b80e0264441fa9026c5296faa8d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:10 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:51344
I20260925 02:50:10.674633 4471 tablet_replica.cc:330] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4: stopping tablet replica
I20260925 02:50:10.675889 4471 raft_consensus.cc:2229] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:50:10.677515 4471 raft_consensus.cc:2258] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:10.679625 4471 ts_tablet_manager.cc:1905] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:10.679669 4469 tablet_replica.cc:330] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d: stopping tablet replica
I20260925 02:50:10.679872 4469 raft_consensus.cc:2229] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:10.680330 4469 raft_consensus.cc:2258] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:10.681665 4469 ts_tablet_manager.cc:1905] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:10.687878 4468 tablet_replica.cc:330] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6: stopping tablet replica
I20260925 02:50:10.688189 4468 raft_consensus.cc:2229] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:10.688607 4468 raft_consensus.cc:2258] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:10.689610 4471 ts_tablet_manager.cc:1918] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:50:10.689821 4471 log.cc:1196] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/076b80e0264441fa9026c5296faa8d89
I20260925 02:50:10.690093 4468 ts_tablet_manager.cc:1905] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:10.690439 4471 ts_tablet_manager.cc:1939] T 076b80e0264441fa9026c5296faa8d89 P fac5091b06704f17b5fe94b423a033b4: Deleting consensus metadata
I20260925 02:50:10.691757 3877 catalog_manager.cc:4841] TS fac5091b06704f17b5fe94b423a033b4 (127.2.172.65:43533): tablet 076b80e0264441fa9026c5296faa8d89 (table test-table [id=eabddb86d25c4d398999dd1c3e768202]) successfully deleted
I20260925 02:50:10.696823 4468 ts_tablet_manager.cc:1918] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:50:10.697129 4468 log.cc:1196] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/076b80e0264441fa9026c5296faa8d89
I20260925 02:50:10.697573 4469 ts_tablet_manager.cc:1918] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:50:10.697746 4469 log.cc:1196] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/076b80e0264441fa9026c5296faa8d89
I20260925 02:50:10.697871 4468 ts_tablet_manager.cc:1939] T 076b80e0264441fa9026c5296faa8d89 P 86514a238b3e4b299c385f16229980b6: Deleting consensus metadata
I20260925 02:50:10.698215 4469 ts_tablet_manager.cc:1939] T 076b80e0264441fa9026c5296faa8d89 P a04284922c3c47248e9d97d8b2ad413d: Deleting consensus metadata
I20260925 02:50:10.699332 3876 catalog_manager.cc:4841] TS 86514a238b3e4b299c385f16229980b6 (127.2.172.66:34735): tablet 076b80e0264441fa9026c5296faa8d89 (table test-table [id=eabddb86d25c4d398999dd1c3e768202]) successfully deleted
I20260925 02:50:10.699347 3875 catalog_manager.cc:4841] TS a04284922c3c47248e9d97d8b2ad413d (127.2.172.67:38389): tablet 076b80e0264441fa9026c5296faa8d89 (table test-table [id=eabddb86d25c4d398999dd1c3e768202]) successfully deleted
Sep 25 02:50:10 dist-test-slave-g1xh krb5kdc[3846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:10.708006 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:10.684750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.2.172.65:43533)
0925 02:50:10.684939 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:10.684951 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:10.688690 (+ 3739us) server_negotiation.cc:407] Connection header received
0925 02:50:10.689177 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:10.689184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:10.689293 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:10.689375 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:10.691837 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.693057 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.695148 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.695913 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.696073 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:10.701541 (+ 5468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:10.701560 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:10.701577 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:10.701621 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:10.704839 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:10.705679 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:10.705686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:10.705689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:10.705856 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:10.706280 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:10.706284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:10.706286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:10.706683 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:10.707030 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:10.707383 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 02:50:10.707666 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 02:50:10.708904 4096 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51362
I20260925 02:50:10.733865 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:10.712103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45252 (local address 127.2.172.126:37161)
0925 02:50:10.718187 (+ 6084us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:10.718192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:10.718215 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:50:10.718274 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:10.718278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:10.718347 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:10.718454 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:10.720413 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.721490 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.723449 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.724033 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.724147 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:10.727311 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:10.727317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:10.727320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:10.727372 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:10.730778 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:10.731617 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:10.731624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:10.731628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:10.731798 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:10.732177 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:10.732183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:10.732186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:10.732574 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:10.732834 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:10.733190 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 02:50:10.733452 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5889,"thread_start_us":5731,"threads_started":1}
W20260925 02:50:10.734513 3892 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45252
I20260925 02:50:10.736532 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 3964
I20260925 02:50:10.757937 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:10.738361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.2.172.65:43533)
0925 02:50:10.738663 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:10.738672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:10.744145 (+ 5473us) server_negotiation.cc:407] Connection header received
0925 02:50:10.744360 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:10.744364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:10.744444 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:10.744556 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:10.745777 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.746554 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.747847 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.748448 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.748606 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:10.751597 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:10.751604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:10.751609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:10.751654 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:10.755082 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:10.755891 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:10.755897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:10.755901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:10.756092 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:10.756486 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:10.756492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:10.756495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:10.756752 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:10.757024 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:10.757323 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 02:50:10.757596 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260925 02:50:10.758796 4096 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51366
W20260925 02:50:10.759536 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 3964: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:50:10.759780 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 3964
I20260925 02:50:10.786477 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 4105
Sep 25 02:50:10 dist-test-slave-g1xh krb5kdc[3846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:50:10.803146 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:10.787302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.2.172.66:34735)
0925 02:50:10.787945 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:10.787950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:10.787986 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:50:10.788045 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:10.788049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:10.788118 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:10.788217 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:10.789712 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.790492 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.792136 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.792571 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.792667 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:10.796914 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:10.796931 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:10.796948 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:10.797006 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:10.800161 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:10.800938 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:10.800946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:10.800949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:10.801081 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:10.801500 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:10.801504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:10.801506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:10.801876 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:10.802204 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:10.802517 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 02:50:10.802765 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":385,"threads_started":1}
W20260925 02:50:10.804008 4237 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57654
W20260925 02:50:10.804800 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 4105: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:50:10.804991 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 4105
I20260925 02:50:10.824195 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 4246
Sep 25 02:50:10 dist-test-slave-g1xh krb5kdc[3846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:50:10.841185 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:10.825126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.2.172.67:38389)
0925 02:50:10.825643 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:10.825648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:10.825665 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:50:10.825727 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:10.825731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:10.825796 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:10.825897 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:10.827163 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.827994 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.829257 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:10.829756 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:10.829970 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:10.834653 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:10.834670 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:10.834688 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:10.834729 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:10.837985 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:10.838874 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:10.838882 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:10.838884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:10.839038 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:10.839580 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:10.839586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:10.839590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:10.840020 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:10.840265 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:10.840584 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 02:50:10.840770 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":236,"threads_started":1}
W20260925 02:50:10.841915 4382 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43044
W20260925 02:50:10.842625 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 4246: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:50:10.842764 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 4246
I20260925 02:50:10.861611 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 3859
I20260925 02:50:10.862815 3932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:10.981539 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 3859
2026-09-25T02:50:11Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5534 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:11 dist-test-slave-g1xh krb5kdc[4482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:11 dist-test-slave-g1xh krb5kdc[4482](info): set up 2 sockets
Sep 25 02:50:11 dist-test-slave-g1xh krb5kdc[4482](info): commencing operation
krb5kdc: starting...
W20260925 02:50:12.138346 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:50:12 dist-test-slave-g1xh krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:50:12.351353 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:43669
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:43669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:12.711071 4495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:12.711443 4495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:12.711643 4495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:12.718858 4495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:12.719014 4495 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:50:12.719111 4495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:12.719202 4495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:12.719295 4495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:12.719381 4495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:12.729478 4495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:43669
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:43669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:12.730948 4495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:12.732461 4495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:12.744798 4500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:12.744827 4501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:12.744850 4503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:12.745601 4495 server_base.cc:1034] running on GCE node
I20260925 02:50:12.746382 4495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:12.747884 4495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:12.749111 4495 hybrid_clock.cc:648] HybridClock initialized: now 1790304612749051 us; error 44 us; skew 500 ppm
I20260925 02:50:12.749470 4495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:12 dist-test-slave-g1xh krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304612, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:12.752682 4495 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:50:12.754073 4495 webserver.cc:460] Webserver started at http://127.2.172.126:36157/ using document root <none> and password file <none>
I20260925 02:50:12.754794 4495 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:12.754881 4495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:12.755151 4495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:12.758184 4495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "f6f3d33331a340429be35f7dbc86bcfe"
format_stamp: "Formatted at 2026-09-25 02:50:12 on dist-test-slave-g1xh"
server_key: "1cf866f766e46c75cfc3cb18a14081b7"
server_key_iv: "d746b2d282198c4d5d0a6e41c34cd7a5"
server_key_version: "encryptionkey@0"
I20260925 02:50:12.758889 4495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "f6f3d33331a340429be35f7dbc86bcfe"
format_stamp: "Formatted at 2026-09-25 02:50:12 on dist-test-slave-g1xh"
server_key: "1cf866f766e46c75cfc3cb18a14081b7"
server_key_iv: "d746b2d282198c4d5d0a6e41c34cd7a5"
server_key_version: "encryptionkey@0"
I20260925 02:50:12.764199 4495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 02:50:12.767930 4511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:12.769583 4495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:50:12.769804 4495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "f6f3d33331a340429be35f7dbc86bcfe"
format_stamp: "Formatted at 2026-09-25 02:50:12 on dist-test-slave-g1xh"
server_key: "1cf866f766e46c75cfc3cb18a14081b7"
server_key_iv: "d746b2d282198c4d5d0a6e41c34cd7a5"
server_key_version: "encryptionkey@0"
I20260925 02:50:12.770085 4495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:12.808791 4495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:12.813292 4495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:12.813591 4495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:12.835733 4495 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:43669
I20260925 02:50:12.835714 4574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:43669 every 8 connection(s)
I20260925 02:50:12.837164 4495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:12.841152 4575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:12.842918 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 4495
I20260925 02:50:12.843142 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:12.844053 2737 external_mini_cluster.cc:1428] Setting key 36d24cdd4cce465fe5e9e1328b6aab9d
I20260925 02:50:12.854669 4575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe: Bootstrap starting.
I20260925 02:50:12.859033 4575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe: Neither blocks nor log segments found. Creating new log.
Sep 25 02:50:12 dist-test-slave-g1xh krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:12.860399 4575 log.cc:824] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:12.864105 4575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe: No bootstrap required, opened a new log
I20260925 02:50:12.870108 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:12.846442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47970 (local address 127.2.172.126:43669)
0925 02:50:12.847354 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:12.847366 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:12.847413 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:50:12.849253 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:12.849276 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:12.849678 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:12.850000 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:12.851892 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:12.853290 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:12.855105 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:12.855948 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:12.856105 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:12.861242 (+ 5137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:12.861250 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:12.861266 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:12.861309 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:12.865218 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:12.865937 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:12.865950 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:12.865960 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:12.866181 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:12.866560 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:12.866565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:12.866569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:12.867317 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:12.867716 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:12.868163 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 02:50:12.868664 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":447,"threads_started":1}
I20260925 02:50:12.874366 4575 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f3d33331a340429be35f7dbc86bcfe" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43669 } }
I20260925 02:50:12.874804 4575 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:12.874894 4575 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f3d33331a340429be35f7dbc86bcfe, State: Initialized, Role: FOLLOWER
I20260925 02:50:12.875437 4575 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f3d33331a340429be35f7dbc86bcfe" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43669 } }
I20260925 02:50:12.875710 4575 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:12.875813 4575 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:12.875952 4575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:12.878398 4575 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f3d33331a340429be35f7dbc86bcfe" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43669 } }
I20260925 02:50:12.878873 4575 leader_election.cc:302] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f3d33331a340429be35f7dbc86bcfe; no voters:
I20260925 02:50:12.879522 4575 leader_election.cc:288] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:12.879751 4580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:12.880573 4580 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [term 1 LEADER]: Becoming Leader. State: Replica: f6f3d33331a340429be35f7dbc86bcfe, State: Running, Role: LEADER
I20260925 02:50:12.881242 4580 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f3d33331a340429be35f7dbc86bcfe" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43669 } }
I20260925 02:50:12.881994 4575 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:12.884575 4581 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6f3d33331a340429be35f7dbc86bcfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f3d33331a340429be35f7dbc86bcfe" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43669 } } }
I20260925 02:50:12.884550 4582 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f3d33331a340429be35f7dbc86bcfe. Latest consensus state: current_term: 1 leader_uuid: "f6f3d33331a340429be35f7dbc86bcfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f3d33331a340429be35f7dbc86bcfe" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43669 } } }
I20260925 02:50:12.885051 4582 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:12.885051 4581 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:12.888990 4587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:12.894645 4587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:12.905494 4587 catalog_manager.cc:1347] Generated new cluster ID: a9938e1025314d76aa7d410279be0b04
I20260925 02:50:12.905642 4587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:12.916852 4587 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:12.918928 4587 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:12.926743 4587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe: Generated new TSK 0
I20260925 02:50:12.927790 4587 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:50:12.984117 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43669
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:13.358742 4602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:13.359061 4602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:13.359167 4602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:13.367107 4602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:13.367254 4602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:13.367357 4602 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:50:13.367468 4602 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:50:13.367542 4602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:50:13.378445 4602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.172.126:43669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:13.379951 4602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:13.381343 4602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:13.397593 4607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:13.397657 4609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:13.398464 4611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:13.398599 4602 server_base.cc:1034] running on GCE node
I20260925 02:50:13.399361 4602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:13.400477 4602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:13.401760 4602 hybrid_clock.cc:648] HybridClock initialized: now 1790304613401680 us; error 65 us; skew 500 ppm
I20260925 02:50:13.402155 4602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:13 dist-test-slave-g1xh krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304613, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:13.413429 4602 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:50:13.415047 4602 webserver.cc:460] Webserver started at http://127.2.172.65:34281/ using document root <none> and password file <none>
I20260925 02:50:13.415931 4602 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:13.416049 4602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:13.416317 4602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:13.419361 4602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "8310947b68724400bb7d8107b8b62b46"
format_stamp: "Formatted at 2026-09-25 02:50:13 on dist-test-slave-g1xh"
server_key: "308c90e50f0766f6dc7f8e0f2fca1279"
server_key_iv: "375e23a9d4b167b9a46686c23263fd2f"
server_key_version: "encryptionkey@0"
I20260925 02:50:13.420210 4602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "8310947b68724400bb7d8107b8b62b46"
format_stamp: "Formatted at 2026-09-25 02:50:13 on dist-test-slave-g1xh"
server_key: "308c90e50f0766f6dc7f8e0f2fca1279"
server_key_iv: "375e23a9d4b167b9a46686c23263fd2f"
server_key_version: "encryptionkey@0"
I20260925 02:50:13.426126 4602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:50:13.430612 4618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:13.432379 4602 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260925 02:50:13.432587 4602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "8310947b68724400bb7d8107b8b62b46"
format_stamp: "Formatted at 2026-09-25 02:50:13 on dist-test-slave-g1xh"
server_key: "308c90e50f0766f6dc7f8e0f2fca1279"
server_key_iv: "375e23a9d4b167b9a46686c23263fd2f"
server_key_version: "encryptionkey@0"
I20260925 02:50:13.432806 4602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:13.450167 4602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:13.454968 4602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:13.455304 4602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:13.458364 4602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:13.458541 4602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:13.458688 4602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:13.458788 4602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:50:13 dist-test-slave-g1xh krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304613, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:13.484833 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:13.459023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:36003 (local address 127.2.172.126:43669)
0925 02:50:13.459954 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:13.459962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:13.460990 (+ 1028us) server_negotiation.cc:407] Connection header received
0925 02:50:13.463432 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:13.463441 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:13.463548 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:13.463669 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:13.466463 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:13.467590 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:13.470150 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:13.470596 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:13.470753 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:13.476046 (+ 5293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:13.476055 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:13.476062 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:13.476116 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:13.479662 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:13.481113 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:13.481121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:13.481127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:13.481327 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:13.482041 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:13.482050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:13.482055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:13.482512 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:13.482682 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:13.483868 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0925 02:50:13.484223 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":661,"threads_started":1}
I20260925 02:50:13.485556 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:13.459803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43669 (local address 127.2.172.65:36003)
0925 02:50:13.460781 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:13.460833 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:13.463022 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:13.464327 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:13.464342 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:13.465226 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:13.466227 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:13.466250 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:13.467817 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:13.467820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:13.469904 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:13.469930 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:13.470886 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:13.470894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:13.471139 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:13.471991 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:13.472015 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:13.475731 (+ 3716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:13.479925 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:13.479944 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:13.479971 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:13.480942 (+ 971us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:13.481523 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:13.481537 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:13.481541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:13.481905 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:13.482679 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:13.482688 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:13.483280 (+ 592us) client_negotiation.cc:769] Sending connection context
0925 02:50:13.483729 (+ 449us) client_negotiation.cc:240] Negotiation successful
0925 02:50:13.484254 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":433,"threads_started":1}
I20260925 02:50:13.511649 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:13.493881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:52227 (local address 127.2.172.126:43669)
0925 02:50:13.494121 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:13.494131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:13.495085 (+ 954us) server_negotiation.cc:407] Connection header received
0925 02:50:13.495345 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:13.495350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:13.495448 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:13.495610 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:13.497235 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:13.498096 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:13.500764 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:13.501353 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:13.501547 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:13.504582 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:13.504588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:13.504591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:13.504633 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:13.507824 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:13.508853 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:13.508871 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:13.508877 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:13.509081 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:13.509675 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:13.509680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:13.509685 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:13.510166 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:13.510335 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:13.511091 (+ 756us) server_negotiation.cc:299] Negotiation successful
0925 02:50:13.511317 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:50:13.512382 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:13.494105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43669 (local address 127.2.172.65:52227)
0925 02:50:13.494922 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:13.494950 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:13.495166 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:13.495644 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:13.495651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:13.496486 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:13.496996 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:13.497015 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:13.498266 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:13.498274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:13.500553 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:13.500576 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:13.501663 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:13.501671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:13.501931 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:13.502844 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:13.502879 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:13.504286 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:13.507992 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:13.507999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:13.508005 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:13.508685 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:13.509262 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:13.509271 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:13.509277 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:13.509495 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:13.510418 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:13.510427 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:13.510764 (+ 337us) client_negotiation.cc:769] Sending connection context
0925 02:50:13.510963 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 02:50:13.511879 (+ 916us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":386,"threads_started":1}
I20260925 02:50:13.515700 4602 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:34451
I20260925 02:50:13.515745 4740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:34451 every 8 connection(s)
I20260925 02:50:13.517405 4602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:13.519665 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 4602
I20260925 02:50:13.519881 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:13.520036 4741 heartbeater.cc:347] Connected to a master server at 127.2.172.126:43669
I20260925 02:50:13.520310 4741 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:13.520774 2737 external_mini_cluster.cc:1428] Setting key 1aa6bacf252d4cdcf655a42505e03853
I20260925 02:50:13.521215 4741 heartbeater.cc:511] Master 127.2.172.126:43669 requested a full tablet report, sending...
I20260925 02:50:13.523520 4527 ts_manager.cc:194] Registered new tserver with Master: 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451)
I20260925 02:50:13.525198 4527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:36003
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:50:13.571662 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43669
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:13.922585 4745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:13.922917 4745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:13.923022 4745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:13.930442 4745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:13.930560 4745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:13.930647 4745 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:50:13.930706 4745 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:50:13.930770 4745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:50:13.941210 4745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.172.126:43669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:13.942684 4745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:13.944092 4745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:13.957365 4751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:13.957458 4753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:13.957401 4750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:13.958709 4745 server_base.cc:1034] running on GCE node
I20260925 02:50:13.959395 4745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:13.960515 4745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:13.961735 4745 hybrid_clock.cc:648] HybridClock initialized: now 1790304613961646 us; error 77 us; skew 500 ppm
I20260925 02:50:13.962071 4745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:13 dist-test-slave-g1xh krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304613, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:13.965411 4745 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:50:13.966966 4745 webserver.cc:460] Webserver started at http://127.2.172.66:34797/ using document root <none> and password file <none>
I20260925 02:50:13.967707 4745 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:13.967815 4745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:13.968094 4745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:13.971135 4745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "1e099f19b26f401ba131cee8de480943"
format_stamp: "Formatted at 2026-09-25 02:50:13 on dist-test-slave-g1xh"
server_key: "5d5664ee37f7547126f6e59223962c24"
server_key_iv: "d62b2805079b35bb08818cc649c773c8"
server_key_version: "encryptionkey@0"
I20260925 02:50:13.971987 4745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "1e099f19b26f401ba131cee8de480943"
format_stamp: "Formatted at 2026-09-25 02:50:13 on dist-test-slave-g1xh"
server_key: "5d5664ee37f7547126f6e59223962c24"
server_key_iv: "d62b2805079b35bb08818cc649c773c8"
server_key_version: "encryptionkey@0"
I20260925 02:50:13.977787 4745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 02:50:13.981825 4760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:13.983742 4745 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:50:13.983912 4745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "1e099f19b26f401ba131cee8de480943"
format_stamp: "Formatted at 2026-09-25 02:50:13 on dist-test-slave-g1xh"
server_key: "5d5664ee37f7547126f6e59223962c24"
server_key_iv: "d62b2805079b35bb08818cc649c773c8"
server_key_version: "encryptionkey@0"
I20260925 02:50:13.984114 4745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:14.002076 4745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:14.006363 4745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:14.006626 4745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:14.009653 4745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:14.009778 4745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:14.009927 4745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:14.010021 4745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:50:14 dist-test-slave-g1xh krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304613, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:14.040373 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.010166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:39081 (local address 127.2.172.126:43669)
0925 02:50:14.010396 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:14.010405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:14.012097 (+ 1692us) server_negotiation.cc:407] Connection header received
0925 02:50:14.014886 (+ 2789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:14.014895 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:14.015013 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:14.015101 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:14.018869 (+ 3768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.020250 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.023043 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.023819 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.023971 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.030372 (+ 6401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:14.030383 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:14.030391 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:14.030449 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:14.034382 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:14.035739 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:14.035748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:14.035752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:14.035955 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:14.036529 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:14.036535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:14.036538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:14.036989 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:14.037160 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:14.039772 (+ 2612us) server_negotiation.cc:299] Negotiation successful
0925 02:50:14.040027 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:50:14.040524 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.010863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43669 (local address 127.2.172.66:39081)
0925 02:50:14.011881 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:14.011941 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:14.014453 (+ 2512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:14.015827 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:14.015843 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:14.016843 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:14.018590 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.018629 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.020467 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.020478 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.022770 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.022785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.024218 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.024227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.024661 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.025787 (+ 1126us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:14.025812 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:14.030041 (+ 4229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:14.034657 (+ 4616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:14.034674 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:14.034700 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:14.035569 (+ 869us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:14.036088 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:14.036094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:14.036098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:14.036378 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:14.037123 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:14.037133 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:14.037739 (+ 606us) client_negotiation.cc:769] Sending connection context
0925 02:50:14.038116 (+ 377us) client_negotiation.cc:240] Negotiation successful
0925 02:50:14.038733 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":378,"threads_started":1}
I20260925 02:50:14.058805 4745 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:39511
I20260925 02:50:14.058856 4881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:39511 every 8 connection(s)
I20260925 02:50:14.060637 4745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:14.063162 4882 heartbeater.cc:347] Connected to a master server at 127.2.172.126:43669
I20260925 02:50:14.063455 4882 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:14.063513 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 4745
I20260925 02:50:14.063717 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:14.064347 2737 external_mini_cluster.cc:1428] Setting key 777c4ec41ddd7e5b0cdccfb809bc060e
I20260925 02:50:14.064347 4882 heartbeater.cc:511] Master 127.2.172.126:43669 requested a full tablet report, sending...
I20260925 02:50:14.066124 4528 ts_manager.cc:194] Registered new tserver with Master: 1e099f19b26f401ba131cee8de480943 (127.2.172.66:39511)
I20260925 02:50:14.067369 4528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:39081
I20260925 02:50:14.069262 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.051356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:37619 (local address 127.2.172.126:43669)
0925 02:50:14.051628 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:14.051639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:14.052577 (+ 938us) server_negotiation.cc:407] Connection header received
0925 02:50:14.052791 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:14.052799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:14.052906 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:14.053027 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:14.054850 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.056156 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.058419 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.059124 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.059250 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.062245 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:14.062254 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:14.062261 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:14.062309 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:14.065769 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:14.066633 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:14.066639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:14.066642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:14.066810 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:14.067238 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:14.067244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:14.067247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:14.067760 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:14.067934 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:14.068550 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 02:50:14.068811 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:50:14.070448 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.051270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43669 (local address 127.2.172.66:37619)
0925 02:50:14.052408 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:14.052453 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:14.052631 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:14.053244 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:14.053252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:14.054094 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:14.054623 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.054643 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.056332 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.056344 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.058213 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.058239 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.059337 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.059345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.059584 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.060294 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:14.060316 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:14.061665 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:14.065948 (+ 4283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:14.065953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:14.065957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:14.066484 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:14.066964 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:14.066970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:14.066974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:14.067114 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:14.068009 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:14.068017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:14.068248 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 02:50:14.069491 (+ 1243us) client_negotiation.cc:240] Negotiation successful
0925 02:50:14.069896 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":714,"threads_started":1}
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:50:14.111367 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43669
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:14.460181 4886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:14.460556 4886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:14.460755 4886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:14.468335 4886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:14.468523 4886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:14.468683 4886 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:50:14.468772 4886 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:50:14.468899 4886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:50:14.480020 4886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.172.126:43669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:14.481489 4886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:14.482978 4886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:14.498354 4891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:14.498384 4894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:14.498418 4892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:14.499155 4886 server_base.cc:1034] running on GCE node
I20260925 02:50:14.499927 4886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:14.501000 4886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:14.502254 4886 hybrid_clock.cc:648] HybridClock initialized: now 1790304614502208 us; error 39 us; skew 500 ppm
I20260925 02:50:14.502650 4886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:14 dist-test-slave-g1xh krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304614, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:14.505968 4886 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:50:14.507630 4886 webserver.cc:460] Webserver started at http://127.2.172.67:38257/ using document root <none> and password file <none>
I20260925 02:50:14.508358 4886 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:14.508515 4886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:14.508839 4886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:14.512250 4886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "621eca29e73c4d0e819db2ec626df1d1"
format_stamp: "Formatted at 2026-09-25 02:50:14 on dist-test-slave-g1xh"
server_key: "125d2e81ac8fc80f759cf1fb3ef40ae1"
server_key_iv: "45c8d85f3f4e7576bbaa280568482b9b"
server_key_version: "encryptionkey@0"
I20260925 02:50:14.513190 4886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "621eca29e73c4d0e819db2ec626df1d1"
format_stamp: "Formatted at 2026-09-25 02:50:14 on dist-test-slave-g1xh"
server_key: "125d2e81ac8fc80f759cf1fb3ef40ae1"
server_key_iv: "45c8d85f3f4e7576bbaa280568482b9b"
server_key_version: "encryptionkey@0"
I20260925 02:50:14.519075 4886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:50:14.522969 4902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:14.524526 4886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 02:50:14.524748 4886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "621eca29e73c4d0e819db2ec626df1d1"
format_stamp: "Formatted at 2026-09-25 02:50:14 on dist-test-slave-g1xh"
server_key: "125d2e81ac8fc80f759cf1fb3ef40ae1"
server_key_iv: "45c8d85f3f4e7576bbaa280568482b9b"
server_key_version: "encryptionkey@0"
I20260925 02:50:14.524981 4886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:14.528820 4741 heartbeater.cc:503] Master 127.2.172.126:43669 was elected leader, sending a full tablet report...
I20260925 02:50:14.543000 4886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:14.547937 4886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:14.548276 4886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:14.551366 4886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:14.551542 4886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:14.552120 4886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:14.552245 4886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:50:14 dist-test-slave-g1xh krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304614, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:14.576543 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.551981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:56147 (local address 127.2.172.126:43669)
0925 02:50:14.552195 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:14.552202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:14.553817 (+ 1615us) server_negotiation.cc:407] Connection header received
0925 02:50:14.555959 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:14.555965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:14.556098 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:14.556218 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:14.558384 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.559499 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.562172 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.563054 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.563231 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.568232 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:14.568242 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:14.568247 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:14.568296 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:14.571809 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:14.573138 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:14.573142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:14.573145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:14.573278 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:14.573924 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:14.573927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:14.573929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:14.574285 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:14.574435 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:14.575747 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0925 02:50:14.576059 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 02:50:14.578934 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.552506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43669 (local address 127.2.172.67:56147)
0925 02:50:14.553590 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:14.553649 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:14.555657 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:14.556517 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:14.556527 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:14.557160 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:14.558157 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.558186 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.559687 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.559695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.561970 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.561991 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.563345 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.563351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.563602 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.564429 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:14.564472 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:14.567850 (+ 3378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:14.571990 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:14.572005 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:14.572035 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:14.572975 (+ 940us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:14.573405 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:14.573414 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:14.573421 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:14.573792 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:14.574443 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:14.574457 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:14.575161 (+ 704us) client_negotiation.cc:769] Sending connection context
0925 02:50:14.575634 (+ 473us) client_negotiation.cc:240] Negotiation successful
0925 02:50:14.576421 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":495,"threads_started":1}
I20260925 02:50:14.605485 4886 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:36135
I20260925 02:50:14.605548 5022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:36135 every 8 connection(s)
I20260925 02:50:14.606207 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.586188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:49279 (local address 127.2.172.126:43669)
0925 02:50:14.586433 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:14.586440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:14.587441 (+ 1001us) server_negotiation.cc:407] Connection header received
0925 02:50:14.587777 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:14.587786 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:14.587917 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:14.588045 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:14.589388 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.590686 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.593636 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.594538 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.594711 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.597765 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:14.597775 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:14.597783 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:14.597840 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:14.602534 (+ 4694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:14.603787 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:14.603792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:14.603794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:14.603946 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:14.604463 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:14.604514 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:14.604517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:14.604900 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:14.605085 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:14.605664 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 02:50:14.605926 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 02:50:14.606361 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.586361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43669 (local address 127.2.172.67:49279)
0925 02:50:14.587287 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:14.587309 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:14.587522 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:14.588120 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:14.588127 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:14.588862 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:14.589203 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.589215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.590867 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.590873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.593347 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.593374 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.594826 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.594834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.595087 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.595933 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:14.595958 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:14.597488 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:14.602765 (+ 5277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:14.602774 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:14.602782 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:14.603640 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:14.604084 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:14.604091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:14.604097 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:14.604302 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:14.605049 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:14.605054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:14.605322 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 02:50:14.605503 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 02:50:14.605879 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":491,"threads_started":1}
I20260925 02:50:14.607231 4886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:14.609670 5023 heartbeater.cc:347] Connected to a master server at 127.2.172.126:43669
I20260925 02:50:14.610001 5023 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:14.610723 5023 heartbeater.cc:511] Master 127.2.172.126:43669 requested a full tablet report, sending...
I20260925 02:50:14.612146 4528 ts_manager.cc:194] Registered new tserver with Master: 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67:36135)
I20260925 02:50:14.612764 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 4886
I20260925 02:50:14.612962 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:14.613410 4528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:56147
I20260925 02:50:14.613638 2737 external_mini_cluster.cc:1428] Setting key 387704ab86a5e2255fb6dbd114de20cb
I20260925 02:50:14.616963 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:50:14 dist-test-slave-g1xh krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:50:14 dist-test-slave-g1xh krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:14.670473 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.655146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.2.172.126:43669)
0925 02:50:14.655343 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:14.655348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:14.655851 (+ 503us) server_negotiation.cc:407] Connection header received
0925 02:50:14.656022 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:14.656026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:14.656098 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:14.656194 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:14.657760 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.658593 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.660056 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.660547 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.660656 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.665060 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:14.665066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:14.665070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:14.665116 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:14.667965 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:14.668697 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:14.668701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:14.668704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:14.668838 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:14.669179 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:14.669182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:14.669184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:14.669475 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:14.669689 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:14.669958 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 02:50:14.670162 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:50:14.675280 4528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:14.683413 4528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:14.714382 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.697070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:36135 (local address 127.0.0.1:43692)
0925 02:50:14.697939 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:14.697977 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:14.698147 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:14.706426 (+ 8279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:14.706434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:14.706483 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:14.707101 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.707113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.708947 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.708952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.711394 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.711406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.713304 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.713310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.713511 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.713582 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 02:50:14.713702 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 02:50:14.713937 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":62,"thread_start_us":458,"threads_started":1}
I20260925 02:50:14.714469 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.697216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:34451 (local address 127.0.0.1:57538)
0925 02:50:14.700424 (+ 3208us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:14.700447 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:14.700586 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:14.706338 (+ 5752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:14.706345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:14.706379 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:14.706822 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.706847 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.709015 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.709020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.711926 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.711943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.713574 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.713580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.713762 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.713808 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:50:14.713900 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 02:50:14.714041 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3047,"mutex_wait_us":2,"thread_start_us":8425,"threads_started":1}
I20260925 02:50:14.715167 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.697652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57538 (local address 127.2.172.65:34451)
0925 02:50:14.704185 (+ 6533us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:14.704199 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:14.704242 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:50:14.704387 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:14.704400 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:14.704761 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:14.707125 (+ 2364us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:14.707339 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.708849 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.712121 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.713371 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.713532 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.713765 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:14.714683 (+ 918us) server_negotiation.cc:299] Negotiation successful
0925 02:50:14.714862 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6279,"thread_start_us":6122,"threads_started":1}
I20260925 02:50:14.715197 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.697229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.2.172.67:36135)
0925 02:50:14.705634 (+ 8405us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:14.705647 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:14.705687 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:50:14.705816 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:14.705827 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:14.706140 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:14.706848 (+ 708us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:14.707262 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.708781 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.711609 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.713073 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.713898 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.714139 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:14.714223 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:50:14.714390 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8179,"thread_start_us":7982,"threads_started":1}
I20260925 02:50:14.718173 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.699643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39511 (local address 127.0.0.1:38748)
0925 02:50:14.705951 (+ 6308us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:14.705973 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:14.706099 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:14.708807 (+ 2708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:14.708812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:14.708852 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:14.710076 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.710092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.712048 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.712054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.715188 (+ 3134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.715205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.717452 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.717457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.717581 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.717603 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:50:14.717713 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:50:14.717848 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6144,"thread_start_us":825,"threads_started":1}
I20260925 02:50:14.718247 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.700223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.2.172.66:39511)
0925 02:50:14.708015 (+ 7792us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:14.708023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:14.708050 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:50:14.708144 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:14.708152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:14.708411 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:14.708608 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:14.710308 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.711882 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.715412 (+ 3530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.716672 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.717117 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.717324 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:14.717778 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 02:50:14.717960 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7636,"thread_start_us":7500,"threads_started":1}
I20260925 02:50:14.719313 4952 tablet_service.cc:1467] Processing CreateTablet for tablet 66b8cd0ce2214a77ac85826d043c3036 (DEFAULT_TABLE table=test-table [id=9b5d6d24623d4c708308aa87b79673a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:14.719313 4669 tablet_service.cc:1467] Processing CreateTablet for tablet 66b8cd0ce2214a77ac85826d043c3036 (DEFAULT_TABLE table=test-table [id=9b5d6d24623d4c708308aa87b79673a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:14.720902 4669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b8cd0ce2214a77ac85826d043c3036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:14.720920 4952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b8cd0ce2214a77ac85826d043c3036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:14.723848 4811 tablet_service.cc:1467] Processing CreateTablet for tablet 66b8cd0ce2214a77ac85826d043c3036 (DEFAULT_TABLE table=test-table [id=9b5d6d24623d4c708308aa87b79673a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:14.725327 4811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b8cd0ce2214a77ac85826d043c3036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:14.736035 5042 tablet_bootstrap.cc:492] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1: Bootstrap starting.
I20260925 02:50:14.738606 5043 tablet_bootstrap.cc:492] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46: Bootstrap starting.
I20260925 02:50:14.741907 5042 tablet_bootstrap.cc:654] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:14.743007 5044 tablet_bootstrap.cc:492] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943: Bootstrap starting.
I20260925 02:50:14.743782 5042 log.cc:824] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:14.743840 5043 tablet_bootstrap.cc:654] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:14.745292 5043 log.cc:824] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:14.747015 5044 tablet_bootstrap.cc:654] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:14.748286 5044 log.cc:824] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:14.751098 5043 tablet_bootstrap.cc:492] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46: No bootstrap required, opened a new log
I20260925 02:50:14.751422 5043 ts_tablet_manager.cc:1397] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260925 02:50:14.752256 5044 tablet_bootstrap.cc:492] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943: No bootstrap required, opened a new log
I20260925 02:50:14.752512 5042 tablet_bootstrap.cc:492] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1: No bootstrap required, opened a new log
I20260925 02:50:14.752524 5044 ts_tablet_manager.cc:1397] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 02:50:14.752787 5042 ts_tablet_manager.cc:1397] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.008s
I20260925 02:50:14.760696 5042 raft_consensus.cc:348] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } }
I20260925 02:50:14.761143 5042 raft_consensus.cc:374] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:14.761243 5042 raft_consensus.cc:729] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 621eca29e73c4d0e819db2ec626df1d1, State: Initialized, Role: FOLLOWER
I20260925 02:50:14.761861 5042 consensus_queue.cc:260] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } }
I20260925 02:50:14.763015 5023 heartbeater.cc:503] Master 127.2.172.126:43669 was elected leader, sending a full tablet report...
I20260925 02:50:14.762789 5043 raft_consensus.cc:348] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } }
I20260925 02:50:14.763232 5043 raft_consensus.cc:374] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:14.763329 5043 raft_consensus.cc:729] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8310947b68724400bb7d8107b8b62b46, State: Initialized, Role: FOLLOWER
I20260925 02:50:14.763347 5044 raft_consensus.cc:348] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } }
I20260925 02:50:14.763743 5042 ts_tablet_manager.cc:1428] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 02:50:14.763785 5044 raft_consensus.cc:374] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:14.763876 5044 raft_consensus.cc:729] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e099f19b26f401ba131cee8de480943, State: Initialized, Role: FOLLOWER
I20260925 02:50:14.764076 5043 consensus_queue.cc:260] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } }
I20260925 02:50:14.764516 5044 consensus_queue.cc:260] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } }
I20260925 02:50:14.766045 5043 ts_tablet_manager.cc:1428] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260925 02:50:14.766991 4882 heartbeater.cc:503] Master 127.2.172.126:43669 was elected leader, sending a full tablet report...
I20260925 02:50:14.767649 5044 ts_tablet_manager.cc:1428] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260925 02:50:14.769977 4742 tablet.cc:2369] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:14.788847 5051 raft_consensus.cc:482] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:14.789256 5051 raft_consensus.cc:504] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } }
I20260925 02:50:14.791114 5051 leader_election.cc:288] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67:36135), 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451)
I20260925 02:50:14.798245 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.791695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:58017 (local address 127.2.172.65:34451)
0925 02:50:14.791936 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:14.791944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:14.792708 (+ 764us) server_negotiation.cc:407] Connection header received
0925 02:50:14.792903 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:14.792908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:14.792966 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:14.793065 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:14.793739 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.794481 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.796502 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.797290 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.797485 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.797566 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:14.797861 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 02:50:14.797997 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:50:14.798388 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.791668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:38025 (local address 127.2.172.67:36135)
0925 02:50:14.791922 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:14.791931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:14.792619 (+ 688us) server_negotiation.cc:407] Connection header received
0925 02:50:14.792824 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:14.792829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:14.792897 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:14.793004 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:14.793628 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.794521 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.796572 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.797430 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.797633 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.797709 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:14.798007 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 02:50:14.798131 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:50:14.798727 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.791658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:34451 (local address 127.2.172.66:58017)
0925 02:50:14.792624 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:14.792639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:14.792745 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:14.793077 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:14.793082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:14.793102 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:14.793582 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.793593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.794619 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.794625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.796332 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.796346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.797509 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.797515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.797701 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.797725 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:50:14.797835 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:50:14.797967 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":28,"thread_start_us":532,"threads_started":1}
I20260925 02:50:14.798779 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.791280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:36135 (local address 127.2.172.66:38025)
0925 02:50:14.792511 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:14.792526 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:14.792695 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:14.793032 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:14.793036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:14.793060 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:14.793480 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.793489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.794654 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.794662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.796414 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.796426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.797662 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.797670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.797828 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.797851 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:50:14.797965 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:50:14.798093 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":40,"thread_start_us":903,"threads_started":1}
I20260925 02:50:14.799928 4978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8cd0ce2214a77ac85826d043c3036" candidate_uuid: "1e099f19b26f401ba131cee8de480943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621eca29e73c4d0e819db2ec626df1d1" is_pre_election: true
I20260925 02:50:14.799922 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8cd0ce2214a77ac85826d043c3036" candidate_uuid: "1e099f19b26f401ba131cee8de480943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8310947b68724400bb7d8107b8b62b46" is_pre_election: true
I20260925 02:50:14.800459 4694 raft_consensus.cc:2454] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e099f19b26f401ba131cee8de480943 in term 0.
I20260925 02:50:14.800452 4978 raft_consensus.cc:2454] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e099f19b26f401ba131cee8de480943 in term 0.
I20260925 02:50:14.801127 4762 leader_election.cc:302] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e099f19b26f401ba131cee8de480943, 8310947b68724400bb7d8107b8b62b46; no voters:
I20260925 02:50:14.801576 5051 raft_consensus.cc:2781] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:14.801738 5051 raft_consensus.cc:482] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:14.801839 5051 raft_consensus.cc:3037] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:14.804122 5051 raft_consensus.cc:504] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } }
I20260925 02:50:14.804764 5051 leader_election.cc:288] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [CANDIDATE]: Term 1 election: Requested vote from peers 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67:36135), 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451)
I20260925 02:50:14.805218 4978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8cd0ce2214a77ac85826d043c3036" candidate_uuid: "1e099f19b26f401ba131cee8de480943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621eca29e73c4d0e819db2ec626df1d1"
I20260925 02:50:14.805284 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8cd0ce2214a77ac85826d043c3036" candidate_uuid: "1e099f19b26f401ba131cee8de480943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8310947b68724400bb7d8107b8b62b46"
I20260925 02:50:14.805441 4978 raft_consensus.cc:3037] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:14.805516 4694 raft_consensus.cc:3037] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:14.807878 4694 raft_consensus.cc:2454] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e099f19b26f401ba131cee8de480943 in term 1.
I20260925 02:50:14.807909 4978 raft_consensus.cc:2454] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e099f19b26f401ba131cee8de480943 in term 1.
I20260925 02:50:14.808326 4762 leader_election.cc:302] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e099f19b26f401ba131cee8de480943, 8310947b68724400bb7d8107b8b62b46; no voters:
I20260925 02:50:14.808635 5051 raft_consensus.cc:2781] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:14.809100 5051 raft_consensus.cc:686] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 1 LEADER]: Becoming Leader. State: Replica: 1e099f19b26f401ba131cee8de480943, State: Running, Role: LEADER
I20260925 02:50:14.809582 5051 consensus_queue.cc:237] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } }
W20260925 02:50:14.812914 4883 tablet.cc:2369] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:14.814275 4526 catalog_manager.cc:5495] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e099f19b26f401ba131cee8de480943 (127.2.172.66). New cstate: current_term: 1 leader_uuid: "1e099f19b26f401ba131cee8de480943" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } health_report { overall_health: HEALTHY } } }
W20260925 02:50:14.859145 5024 tablet.cc:2369] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:50:14 dist-test-slave-g1xh krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304612, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:14.876466 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.862290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.2.172.126:43669)
0925 02:50:14.862518 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:14.862522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:14.863031 (+ 509us) server_negotiation.cc:407] Connection header received
0925 02:50:14.863255 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:14.863259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:14.863336 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:14.863449 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:14.864652 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.865395 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.866643 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.867076 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.867214 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.870682 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:14.870687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:14.870691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:14.870719 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:14.873555 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:14.874332 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:14.874338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:14.874343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:14.874486 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:14.874849 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:14.874856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:14.874859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:14.875143 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:14.875296 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:14.875833 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 02:50:14.876049 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 02:50:14.876595 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:14.862306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43669 (local address 127.0.0.1:47990)
0925 02:50:14.862917 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:14.862934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:14.863113 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:14.863478 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:14.863482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:14.864216 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:14.864511 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.864521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.865526 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.865533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.866483 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:14.866494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:14.867254 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:14.867260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:14.867421 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:14.868165 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:14.868183 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:14.870474 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:14.873709 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:14.873727 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:14.873735 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:14.874186 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:14.874601 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:14.874606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:14.874610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:14.874740 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:14.875294 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:14.875304 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:14.875478 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 02:50:14.875745 (+ 267us) client_negotiation.cc:240] Negotiation successful
0925 02:50:14.876176 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":141,"thread_start_us":364,"threads_started":1}
I20260925 02:50:14.882180 4526 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:47990:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:50:14.883404 4526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:14.890988 4811 tablet_service.cc:1467] Processing CreateTablet for tablet 8431dc150d9f4239bb6e9b7de95daea1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=931cd6aeb026423083f2a5d0e9170f8d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:50:14.890789 4669 tablet_service.cc:1467] Processing CreateTablet for tablet 8431dc150d9f4239bb6e9b7de95daea1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=931cd6aeb026423083f2a5d0e9170f8d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:50:14.891836 4811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8431dc150d9f4239bb6e9b7de95daea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:14.891836 4669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8431dc150d9f4239bb6e9b7de95daea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:14.891335 4952 tablet_service.cc:1467] Processing CreateTablet for tablet 8431dc150d9f4239bb6e9b7de95daea1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=931cd6aeb026423083f2a5d0e9170f8d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:50:14.892243 4952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8431dc150d9f4239bb6e9b7de95daea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:14.897239 5044 tablet_bootstrap.cc:492] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943: Bootstrap starting.
I20260925 02:50:14.898672 5044 tablet_bootstrap.cc:654] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:14.899376 5042 tablet_bootstrap.cc:492] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: Bootstrap starting.
I20260925 02:50:14.899438 5043 tablet_bootstrap.cc:492] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46: Bootstrap starting.
I20260925 02:50:14.900816 5042 tablet_bootstrap.cc:654] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:14.901693 5043 tablet_bootstrap.cc:654] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:14.903687 5042 tablet_bootstrap.cc:492] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: No bootstrap required, opened a new log
I20260925 02:50:14.903852 5042 ts_tablet_manager.cc:1397] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:50:14.904438 5043 tablet_bootstrap.cc:492] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46: No bootstrap required, opened a new log
I20260925 02:50:14.904623 5043 ts_tablet_manager.cc:1397] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:50:14.904560 5042 raft_consensus.cc:348] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:14.904820 5042 raft_consensus.cc:374] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:14.904915 5042 raft_consensus.cc:729] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 621eca29e73c4d0e819db2ec626df1d1, State: Initialized, Role: FOLLOWER
I20260925 02:50:14.905159 5042 consensus_queue.cc:260] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:14.905628 5043 raft_consensus.cc:348] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:14.905969 5043 raft_consensus.cc:374] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:14.906080 5043 raft_consensus.cc:729] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8310947b68724400bb7d8107b8b62b46, State: Initialized, Role: FOLLOWER
I20260925 02:50:14.905977 5049 tablet_replica.cc:439] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } }
I20260925 02:50:14.906342 5049 tablet_replica.cc:442] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:14.906837 5042 ts_tablet_manager.cc:1428] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:50:14.906440 5043 consensus_queue.cc:260] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:14.908494 5050 tablet_replica.cc:439] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } }
I20260925 02:50:14.908944 5050 tablet_replica.cc:442] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:14.909518 5044 tablet_bootstrap.cc:492] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943: No bootstrap required, opened a new log
I20260925 02:50:14.910391 5043 ts_tablet_manager.cc:1428] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260925 02:50:14.910629 5044 ts_tablet_manager.cc:1397] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260925 02:50:14.911521 5044 raft_consensus.cc:348] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:14.911823 5044 raft_consensus.cc:374] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:14.911942 5044 raft_consensus.cc:729] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e099f19b26f401ba131cee8de480943, State: Initialized, Role: FOLLOWER
I20260925 02:50:14.912179 5044 consensus_queue.cc:260] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:14.912636 5051 tablet_replica.cc:439] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } }
I20260925 02:50:14.912895 5044 ts_tablet_manager.cc:1428] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:50:14.912976 5051 tablet_replica.cc:442] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:15.086068 5049 raft_consensus.cc:482] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:15.086318 5049 raft_consensus.cc:504] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.088188 5049 leader_election.cc:288] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451), 1e099f19b26f401ba131cee8de480943 (127.2.172.66:39511)
I20260925 02:50:15.096969 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.088558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:34451 (local address 127.2.172.67:32935)
0925 02:50:15.089459 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:15.089482 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:15.089707 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:15.090202 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:15.090208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:15.090244 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:15.090605 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.090618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.092363 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.092369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.094550 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.094565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.096031 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.096037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.096230 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.096257 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:50:15.096383 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 02:50:15.096569 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":728,"threads_started":1}
I20260925 02:50:15.096966 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.089048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:32935 (local address 127.2.172.65:34451)
0925 02:50:15.089310 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:15.089318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:15.089606 (+ 288us) server_negotiation.cc:407] Connection header received
0925 02:50:15.089911 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:15.089920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:15.090031 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:15.090148 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:15.090899 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.092178 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.094727 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.095820 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.096031 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.096121 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:15.096500 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 02:50:15.096659 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:50:15.097287 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.088825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:49889 (local address 127.2.172.66:39511)
0925 02:50:15.089358 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:15.089363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:15.090551 (+ 1188us) server_negotiation.cc:407] Connection header received
0925 02:50:15.090662 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:15.090670 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:15.090775 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:15.090880 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:15.091688 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.092761 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.095038 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.096242 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.096436 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.096513 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:15.096843 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 02:50:15.096997 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260925 02:50:15.097345 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.088874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39511 (local address 127.2.172.67:49889)
0925 02:50:15.090419 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:15.090437 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:15.090549 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:15.090927 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:15.090931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:15.090961 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:15.091446 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.091459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.092961 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.092966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.094853 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.094867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.096475 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.096481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.096665 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.096698 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:50:15.096791 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 02:50:15.096913 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"thread_start_us":1116,"threads_started":1}
I20260925 02:50:15.098467 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8431dc150d9f4239bb6e9b7de95daea1" candidate_uuid: "621eca29e73c4d0e819db2ec626df1d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8310947b68724400bb7d8107b8b62b46" is_pre_election: true
I20260925 02:50:15.098815 4694 raft_consensus.cc:2454] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 621eca29e73c4d0e819db2ec626df1d1 in term 0.
I20260925 02:50:15.098796 4831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8431dc150d9f4239bb6e9b7de95daea1" candidate_uuid: "621eca29e73c4d0e819db2ec626df1d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e099f19b26f401ba131cee8de480943" is_pre_election: true
I20260925 02:50:15.099349 4831 raft_consensus.cc:2454] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 621eca29e73c4d0e819db2ec626df1d1 in term 0.
I20260925 02:50:15.099444 4904 leader_election.cc:302] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 621eca29e73c4d0e819db2ec626df1d1, 8310947b68724400bb7d8107b8b62b46; no voters:
I20260925 02:50:15.100036 5049 raft_consensus.cc:2781] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:15.100179 5049 raft_consensus.cc:482] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:15.100288 5049 raft_consensus.cc:3037] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:15.102725 5049 raft_consensus.cc:504] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.103399 5049 leader_election.cc:288] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [CANDIDATE]: Term 1 election: Requested vote from peers 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451), 1e099f19b26f401ba131cee8de480943 (127.2.172.66:39511)
I20260925 02:50:15.104029 4831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8431dc150d9f4239bb6e9b7de95daea1" candidate_uuid: "621eca29e73c4d0e819db2ec626df1d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e099f19b26f401ba131cee8de480943"
I20260925 02:50:15.104061 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8431dc150d9f4239bb6e9b7de95daea1" candidate_uuid: "621eca29e73c4d0e819db2ec626df1d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8310947b68724400bb7d8107b8b62b46"
I20260925 02:50:15.104307 4831 raft_consensus.cc:3037] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:15.104346 4694 raft_consensus.cc:3037] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:15.106774 4831 raft_consensus.cc:2454] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 621eca29e73c4d0e819db2ec626df1d1 in term 1.
I20260925 02:50:15.106761 4694 raft_consensus.cc:2454] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 621eca29e73c4d0e819db2ec626df1d1 in term 1.
I20260925 02:50:15.107321 4904 leader_election.cc:302] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 621eca29e73c4d0e819db2ec626df1d1, 8310947b68724400bb7d8107b8b62b46; no voters:
I20260925 02:50:15.107728 5049 raft_consensus.cc:2781] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:15.108210 5049 raft_consensus.cc:686] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [term 1 LEADER]: Becoming Leader. State: Replica: 621eca29e73c4d0e819db2ec626df1d1, State: Running, Role: LEADER
I20260925 02:50:15.108800 5049 consensus_queue.cc:237] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.112833 5067 tablet_replica.cc:439] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: TxnStatusTablet state changed. Reason: New leader 621eca29e73c4d0e819db2ec626df1d1. Latest consensus state: current_term: 1 leader_uuid: "621eca29e73c4d0e819db2ec626df1d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } }
I20260925 02:50:15.113137 5067 tablet_replica.cc:442] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:50:15.114180 5069 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:50:15.114507 5069 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:50:15.115350 4526 catalog_manager.cc:5495] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67). New cstate: current_term: 1 leader_uuid: "621eca29e73c4d0e819db2ec626df1d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } health_report { overall_health: HEALTHY } } }
I20260925 02:50:15.158429 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.151799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.2.172.67:36135)
0925 02:50:15.151999 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:15.152003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:15.152106 (+ 103us) server_negotiation.cc:407] Connection header received
0925 02:50:15.152483 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:15.152487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:15.152587 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:15.152813 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:15.153223 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.154316 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.155738 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.156208 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.156356 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.156895 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:15.157617 (+ 722us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:15.157812 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:15.157921 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:50:15.158024 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:50:15.158423 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.151672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:36135 (local address 127.0.0.1:43702)
0925 02:50:15.151968 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:15.151992 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:15.152202 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:15.152830 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:15.152835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:15.152848 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:50:15.153078 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.153089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.154448 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.154456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.155537 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.155601 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.156391 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.156397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.156603 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.156677 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:50:15.157746 (+ 1069us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:50:15.157784 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:50:15.157899 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:50:15.158078 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 02:50:15.169626 4694 raft_consensus.cc:1261] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Refusing update from remote peer 621eca29e73c4d0e819db2ec626df1d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:15.169611 4831 raft_consensus.cc:1261] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Refusing update from remote peer 621eca29e73c4d0e819db2ec626df1d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:15.170801 5067 consensus_queue.cc:1035] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:15.171228 5049 consensus_queue.cc:1035] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:15.180732 5051 tablet_replica.cc:439] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943: TxnStatusTablet state changed. Reason: New leader 621eca29e73c4d0e819db2ec626df1d1. Latest consensus state: current_term: 1 leader_uuid: "621eca29e73c4d0e819db2ec626df1d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } }
I20260925 02:50:15.181269 5051 tablet_replica.cc:442] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:15.188970 5051 tablet_replica.cc:439] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "621eca29e73c4d0e819db2ec626df1d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } }
I20260925 02:50:15.189386 5051 tablet_replica.cc:442] T 8431dc150d9f4239bb6e9b7de95daea1 P 1e099f19b26f401ba131cee8de480943: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:15.190773 5050 tablet_replica.cc:439] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46: TxnStatusTablet state changed. Reason: New leader 621eca29e73c4d0e819db2ec626df1d1. Latest consensus state: current_term: 1 leader_uuid: "621eca29e73c4d0e819db2ec626df1d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } }
I20260925 02:50:15.191140 5050 tablet_replica.cc:442] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:15.193184 5049 tablet_replica.cc:439] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "621eca29e73c4d0e819db2ec626df1d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } }
I20260925 02:50:15.193514 5049 tablet_replica.cc:442] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:50:15.209297 5050 tablet_replica.cc:439] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "621eca29e73c4d0e819db2ec626df1d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } }
I20260925 02:50:15.209229 5067 tablet_replica.cc:439] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "621eca29e73c4d0e819db2ec626df1d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } } }
I20260925 02:50:15.209604 5067 tablet_replica.cc:442] T 8431dc150d9f4239bb6e9b7de95daea1 P 621eca29e73c4d0e819db2ec626df1d1: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:50:15.209602 5050 tablet_replica.cc:442] T 8431dc150d9f4239bb6e9b7de95daea1 P 8310947b68724400bb7d8107b8b62b46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:50:15.221328 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.212866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38764 (local address 127.2.172.66:39511)
0925 02:50:15.213089 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:15.213096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:15.214036 (+ 940us) server_negotiation.cc:407] Connection header received
0925 02:50:15.214284 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:15.214291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:15.214433 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:15.214612 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:15.215283 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.216540 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.218128 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.218877 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.219042 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.219501 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:15.220533 (+ 1032us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:15.220704 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:15.220823 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 02:50:15.220932 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:50:15.242633 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.236247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:36561 (local address 127.2.172.67:36135)
0925 02:50:15.236574 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:15.236582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:15.236875 (+ 293us) server_negotiation.cc:407] Connection header received
0925 02:50:15.237135 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:15.237139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:15.237221 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:15.237328 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:15.237998 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.238926 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.240522 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.241036 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.241203 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.241737 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:15.241879 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:15.241986 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:15.242146 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 02:50:15.242250 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 02:50:15.242725 5086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.235953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:36135 (local address 127.2.172.66:36561)
0925 02:50:15.236762 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:15.236780 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:15.236973 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:15.237349 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:15.237354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:15.237367 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:50:15.237842 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.237857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.239109 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.239115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.240367 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.240380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.241253 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.241261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.241410 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.241484 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:50:15.241996 (+ 512us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:50:15.242011 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:50:15.242123 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:50:15.242293 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":337,"threads_started":1}
I20260925 02:50:15.255285 4694 raft_consensus.cc:1261] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Refusing update from remote peer 1e099f19b26f401ba131cee8de480943: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:15.255649 4978 raft_consensus.cc:1261] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Refusing update from remote peer 1e099f19b26f401ba131cee8de480943: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:15.256122 5054 consensus_queue.cc:1035] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:15.256549 5051 consensus_queue.cc:1035] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:15.262658 5074 mvcc.cc:205] Tried to move back new op lower bound from 7333087704078348288 to 7333087702264381440. Current Snapshot: MvccSnapshot[applied={T|T < 7333087704078348288}]
I20260925 02:50:15.266345 5073 mvcc.cc:205] Tried to move back new op lower bound from 7333087704078348288 to 7333087702264381440. Current Snapshot: MvccSnapshot[applied={T|T < 7333087704078348288}]
I20260925 02:50:15.301530 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.293484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:43059 (local address 127.2.172.66:39511)
0925 02:50:15.293920 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:15.293931 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:15.294614 (+ 683us) server_negotiation.cc:407] Connection header received
0925 02:50:15.294878 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:15.294885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:15.294984 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:15.295090 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:15.295865 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.296780 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.299213 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.299918 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.300007 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.300665 (+ 658us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:15.300800 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:15.300915 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:15.301049 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 02:50:15.301145 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260925 02:50:15.301687 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.293399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39511 (local address 127.2.172.67:43059)
0925 02:50:15.294487 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:15.294512 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:15.294700 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:15.295109 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:15.295114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:15.295127 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:50:15.295621 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.295640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.296935 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.296942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.299028 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.299046 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.300196 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.300201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.300354 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.300424 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:50:15.300914 (+ 490us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:50:15.300934 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:50:15.301025 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:50:15.301201 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":546,"threads_started":1}
I20260925 02:50:15.355707 4527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:15.356122 4527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:15.361666 4527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe: Sending DeleteTablet for 3 replicas of tablet 66b8cd0ce2214a77ac85826d043c3036
I20260925 02:50:15.363276 4952 tablet_service.cc:1514] Processing DeleteTablet for tablet 66b8cd0ce2214a77ac85826d043c3036 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:15 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:43692
I20260925 02:50:15.364725 4669 tablet_service.cc:1514] Processing DeleteTablet for tablet 66b8cd0ce2214a77ac85826d043c3036 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:15 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:57538
I20260925 02:50:15.365033 4811 tablet_service.cc:1514] Processing DeleteTablet for tablet 66b8cd0ce2214a77ac85826d043c3036 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:15 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:38748
I20260925 02:50:15.366079 4527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:15.367007 4527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:15.379906 4668 tablet_service.cc:1467] Processing CreateTablet for tablet e3fe0ca14e8747a1803908b62481c88f (DEFAULT_TABLE table=test-table [id=9941c3ec3546487a826509fe8767324b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:15.380771 4668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3fe0ca14e8747a1803908b62481c88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:15.383725 5096 tablet_replica.cc:330] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943: stopping tablet replica
I20260925 02:50:15.383771 4951 tablet_service.cc:1467] Processing CreateTablet for tablet e3fe0ca14e8747a1803908b62481c88f (DEFAULT_TABLE table=test-table [id=9941c3ec3546487a826509fe8767324b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:15.384404 5096 raft_consensus.cc:2229] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:50:15.384555 4951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3fe0ca14e8747a1803908b62481c88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:15.385388 5096 raft_consensus.cc:2258] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:15.388283 4810 tablet_service.cc:1467] Processing CreateTablet for tablet e3fe0ca14e8747a1803908b62481c88f (DEFAULT_TABLE table=test-table [id=9941c3ec3546487a826509fe8767324b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:15.388464 5096 ts_tablet_manager.cc:1905] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:15.389869 4810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3fe0ca14e8747a1803908b62481c88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:15.398176 5095 tablet_replica.cc:330] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46: stopping tablet replica
I20260925 02:50:15.398918 5095 raft_consensus.cc:2229] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:15.399484 5095 raft_consensus.cc:2258] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:15.402308 5096 ts_tablet_manager.cc:1918] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:50:15.402422 5095 ts_tablet_manager.cc:1905] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:15.402539 5096 log.cc:1196] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/66b8cd0ce2214a77ac85826d043c3036
I20260925 02:50:15.403180 5096 ts_tablet_manager.cc:1939] T 66b8cd0ce2214a77ac85826d043c3036 P 1e099f19b26f401ba131cee8de480943: Deleting consensus metadata
I20260925 02:50:15.409809 4512 catalog_manager.cc:4841] TS 1e099f19b26f401ba131cee8de480943 (127.2.172.66:39511): tablet 66b8cd0ce2214a77ac85826d043c3036 (table test-table [id=9b5d6d24623d4c708308aa87b79673a5]) successfully deleted
I20260925 02:50:15.418078 5098 tablet_bootstrap.cc:492] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46: Bootstrap starting.
I20260925 02:50:15.424856 5098 tablet_bootstrap.cc:654] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:15.428061 5099 tablet_bootstrap.cc:492] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943: Bootstrap starting.
I20260925 02:50:15.429705 5094 tablet_replica.cc:330] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1: stopping tablet replica
I20260925 02:50:15.430380 5099 tablet_bootstrap.cc:654] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:15.430538 5095 ts_tablet_manager.cc:1918] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:50:15.431334 5095 log.cc:1196] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/66b8cd0ce2214a77ac85826d043c3036
I20260925 02:50:15.432076 5095 ts_tablet_manager.cc:1939] T 66b8cd0ce2214a77ac85826d043c3036 P 8310947b68724400bb7d8107b8b62b46: Deleting consensus metadata
I20260925 02:50:15.432367 5094 raft_consensus.cc:2229] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:15.432806 5094 raft_consensus.cc:2258] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:15.435561 4513 catalog_manager.cc:4841] TS 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451): tablet 66b8cd0ce2214a77ac85826d043c3036 (table test-table [id=9b5d6d24623d4c708308aa87b79673a5]) successfully deleted
I20260925 02:50:15.437387 5097 tablet_bootstrap.cc:492] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1: Bootstrap starting.
I20260925 02:50:15.438356 5094 ts_tablet_manager.cc:1905] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:15.439495 5098 tablet_bootstrap.cc:492] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46: No bootstrap required, opened a new log
I20260925 02:50:15.439778 5098 ts_tablet_manager.cc:1397] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20260925 02:50:15.440949 5098 raft_consensus.cc:348] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.441452 5098 raft_consensus.cc:374] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:15.441632 5098 raft_consensus.cc:729] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8310947b68724400bb7d8107b8b62b46, State: Initialized, Role: FOLLOWER
I20260925 02:50:15.442102 5098 consensus_queue.cc:260] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.444352 5097 tablet_bootstrap.cc:654] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:15.448351 5098 ts_tablet_manager.cc:1428] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20260925 02:50:15.454100 5094 ts_tablet_manager.cc:1918] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:50:15.454339 5094 log.cc:1196] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/66b8cd0ce2214a77ac85826d043c3036
I20260925 02:50:15.454988 5097 tablet_bootstrap.cc:492] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1: No bootstrap required, opened a new log
I20260925 02:50:15.455027 5094 ts_tablet_manager.cc:1939] T 66b8cd0ce2214a77ac85826d043c3036 P 621eca29e73c4d0e819db2ec626df1d1: Deleting consensus metadata
I20260925 02:50:15.455343 5097 ts_tablet_manager.cc:1397] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20260925 02:50:15.457665 4515 catalog_manager.cc:4841] TS 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67:36135): tablet 66b8cd0ce2214a77ac85826d043c3036 (table test-table [id=9b5d6d24623d4c708308aa87b79673a5]) successfully deleted
I20260925 02:50:15.456722 5097 raft_consensus.cc:348] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.459091 5097 raft_consensus.cc:374] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:15.459278 5099 tablet_bootstrap.cc:492] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943: No bootstrap required, opened a new log
I20260925 02:50:15.459280 5097 raft_consensus.cc:729] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 621eca29e73c4d0e819db2ec626df1d1, State: Initialized, Role: FOLLOWER
I20260925 02:50:15.459518 5099 ts_tablet_manager.cc:1397] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.006s
I20260925 02:50:15.459748 5097 consensus_queue.cc:260] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.460752 5097 ts_tablet_manager.cc:1428] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260925 02:50:15.460702 5099 raft_consensus.cc:348] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.461145 5099 raft_consensus.cc:374] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:15.461328 5099 raft_consensus.cc:729] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e099f19b26f401ba131cee8de480943, State: Initialized, Role: FOLLOWER
I20260925 02:50:15.461746 5099 consensus_queue.cc:260] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.462620 5099 ts_tablet_manager.cc:1428] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:50:15.557004 5050 raft_consensus.cc:482] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:15.557379 5050 raft_consensus.cc:504] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.561724 5050 leader_election.cc:288] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e099f19b26f401ba131cee8de480943 (127.2.172.66:39511), 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67:36135)
I20260925 02:50:15.565793 5051 raft_consensus.cc:482] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:15.566078 5051 raft_consensus.cc:504] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.567268 5051 leader_election.cc:288] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451), 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67:36135)
I20260925 02:50:15.568491 4978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fe0ca14e8747a1803908b62481c88f" candidate_uuid: "1e099f19b26f401ba131cee8de480943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621eca29e73c4d0e819db2ec626df1d1" is_pre_election: true
I20260925 02:50:15.568890 4978 raft_consensus.cc:2454] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e099f19b26f401ba131cee8de480943 in term 0.
I20260925 02:50:15.569962 4764 leader_election.cc:302] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e099f19b26f401ba131cee8de480943, 621eca29e73c4d0e819db2ec626df1d1; no voters:
I20260925 02:50:15.570461 5051 raft_consensus.cc:2781] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:15.570631 5051 raft_consensus.cc:482] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:15.570775 5051 raft_consensus.cc:3037] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:15.577258 5051 raft_consensus.cc:504] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.582203 4978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fe0ca14e8747a1803908b62481c88f" candidate_uuid: "1e099f19b26f401ba131cee8de480943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621eca29e73c4d0e819db2ec626df1d1"
I20260925 02:50:15.582634 4978 raft_consensus.cc:3037] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:15.586899 4978 raft_consensus.cc:2454] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e099f19b26f401ba131cee8de480943 in term 1.
I20260925 02:50:15.587844 4764 leader_election.cc:302] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e099f19b26f401ba131cee8de480943, 621eca29e73c4d0e819db2ec626df1d1; no voters:
I20260925 02:50:15.588587 5092 raft_consensus.cc:2781] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:15.588837 5092 raft_consensus.cc:686] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 1 LEADER]: Becoming Leader. State: Replica: 1e099f19b26f401ba131cee8de480943, State: Running, Role: LEADER
I20260925 02:50:15.589296 5092 consensus_queue.cc:237] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:15.590431 5104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.561148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39511 (local address 127.2.172.65:45273)
0925 02:50:15.577306 (+ 16158us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:15.577342 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:15.577629 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:15.578882 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:15.578892 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:15.578935 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:15.579736 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.579758 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.582001 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.582007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.585055 (+ 3048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.585082 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.587927 (+ 2845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.587935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.588558 (+ 623us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.588620 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 02:50:15.589533 (+ 913us) client_negotiation.cc:240] Negotiation successful
0925 02:50:15.589751 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15900,"thread_start_us":16580,"threads_started":1}
I20260925 02:50:15.591710 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.561209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:45273 (local address 127.2.172.66:39511)
0925 02:50:15.561457 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:15.561464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:15.577493 (+ 16029us) server_negotiation.cc:407] Connection header received
0925 02:50:15.578290 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:15.578303 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:15.578449 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:15.578551 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:15.579995 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.581703 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.585329 (+ 3626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.587066 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.587303 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.587444 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:15.589215 (+ 1771us) server_negotiation.cc:299] Negotiation successful
0925 02:50:15.589415 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 02:50:15.592515 4831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fe0ca14e8747a1803908b62481c88f" candidate_uuid: "8310947b68724400bb7d8107b8b62b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e099f19b26f401ba131cee8de480943" is_pre_election: true
I20260925 02:50:15.601900 4527 catalog_manager.cc:5495] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e099f19b26f401ba131cee8de480943 (127.2.172.66). New cstate: current_term: 1 leader_uuid: "1e099f19b26f401ba131cee8de480943" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } health_report { overall_health: UNKNOWN } } }
I20260925 02:50:15.607199 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.562546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:44555 (local address 127.2.172.67:36135)
0925 02:50:15.562869 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:15.562880 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:15.591405 (+ 28525us) server_negotiation.cc:407] Connection header received
0925 02:50:15.591798 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:15.591806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:15.591918 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:15.592010 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:15.593047 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.594857 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.603972 (+ 9115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.605628 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.605809 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.605918 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:15.606635 (+ 717us) server_negotiation.cc:299] Negotiation successful
0925 02:50:15.606827 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 02:50:15.608804 5104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:15.561980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:36135 (local address 127.2.172.65:44555)
0925 02:50:15.591286 (+ 29306us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:15.591315 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:15.591573 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:15.592504 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:15.592512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:15.592548 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:15.592841 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.592857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.595793 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.595803 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.603666 (+ 7863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:15.603698 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:15.606131 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:15.606138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:15.606374 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:15.606431 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 02:50:15.608224 (+ 1793us) client_negotiation.cc:240] Negotiation successful
0925 02:50:15.608450 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29128,"threads_started":1}
I20260925 02:50:15.588850 5051 leader_election.cc:288] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [CANDIDATE]: Term 1 election: Requested vote from peers 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451), 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67:36135)
I20260925 02:50:15.620563 4690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fe0ca14e8747a1803908b62481c88f" candidate_uuid: "1e099f19b26f401ba131cee8de480943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8310947b68724400bb7d8107b8b62b46" is_pre_election: true
I20260925 02:50:15.620957 4690 raft_consensus.cc:2454] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e099f19b26f401ba131cee8de480943 in term 0.
I20260925 02:50:15.621181 4978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fe0ca14e8747a1803908b62481c88f" candidate_uuid: "8310947b68724400bb7d8107b8b62b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621eca29e73c4d0e819db2ec626df1d1" is_pre_election: true
I20260925 02:50:15.621415 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fe0ca14e8747a1803908b62481c88f" candidate_uuid: "1e099f19b26f401ba131cee8de480943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8310947b68724400bb7d8107b8b62b46"
I20260925 02:50:15.621630 4694 raft_consensus.cc:3037] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:15.621680 4978 raft_consensus.cc:2379] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8310947b68724400bb7d8107b8b62b46 in current term 1: Already voted for candidate 1e099f19b26f401ba131cee8de480943 in this term.
I20260925 02:50:15.625404 4694 raft_consensus.cc:2454] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e099f19b26f401ba131cee8de480943 in term 1.
I20260925 02:50:15.626058 4622 leader_election.cc:302] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8310947b68724400bb7d8107b8b62b46; no voters: 1e099f19b26f401ba131cee8de480943, 621eca29e73c4d0e819db2ec626df1d1
I20260925 02:50:15.626633 5050 raft_consensus.cc:2726] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 02:50:15.682698 4978 raft_consensus.cc:1261] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Refusing update from remote peer 1e099f19b26f401ba131cee8de480943: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:15.683667 5092 consensus_queue.cc:1035] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:15.683849 4694 raft_consensus.cc:1261] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Refusing update from remote peer 1e099f19b26f401ba131cee8de480943: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:15.684502 5092 consensus_queue.cc:1035] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:50:15.731879 5074 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:50:17.344437 4526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:17.344707 4526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:17.350267 4526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe: Sending DeleteTablet for 3 replicas of tablet e3fe0ca14e8747a1803908b62481c88f
I20260925 02:50:17.351503 4810 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fe0ca14e8747a1803908b62481c88f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:17 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:38748
I20260925 02:50:17.351480 4951 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fe0ca14e8747a1803908b62481c88f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:17 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:43692
I20260925 02:50:17.351475 4668 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fe0ca14e8747a1803908b62481c88f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:17 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:57538
I20260925 02:50:17.352499 5133 tablet_replica.cc:330] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46: stopping tablet replica
I20260925 02:50:17.352785 5133 raft_consensus.cc:2229] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:17.353137 5133 raft_consensus.cc:2258] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:17.353816 5132 tablet_replica.cc:330] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943: stopping tablet replica
I20260925 02:50:17.354055 5132 raft_consensus.cc:2229] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:50:17.354528 5132 raft_consensus.cc:2258] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:17.355073 5133 ts_tablet_manager.cc:1905] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:17.355175 5131 tablet_replica.cc:330] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1: stopping tablet replica
I20260925 02:50:17.355406 5131 raft_consensus.cc:2229] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:17.355703 5131 raft_consensus.cc:2258] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:17.355824 5132 ts_tablet_manager.cc:1905] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:17.356827 5131 ts_tablet_manager.cc:1905] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:17.363412 5133 ts_tablet_manager.cc:1918] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:50:17.363643 5133 log.cc:1196] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/e3fe0ca14e8747a1803908b62481c88f
I20260925 02:50:17.364056 5131 ts_tablet_manager.cc:1918] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:50:17.364073 5132 ts_tablet_manager.cc:1918] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:50:17.364118 5133 ts_tablet_manager.cc:1939] T e3fe0ca14e8747a1803908b62481c88f P 8310947b68724400bb7d8107b8b62b46: Deleting consensus metadata
I20260925 02:50:17.364234 5131 log.cc:1196] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/e3fe0ca14e8747a1803908b62481c88f
I20260925 02:50:17.364234 5132 log.cc:1196] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/e3fe0ca14e8747a1803908b62481c88f
I20260925 02:50:17.364770 5132 ts_tablet_manager.cc:1939] T e3fe0ca14e8747a1803908b62481c88f P 1e099f19b26f401ba131cee8de480943: Deleting consensus metadata
I20260925 02:50:17.364794 5131 ts_tablet_manager.cc:1939] T e3fe0ca14e8747a1803908b62481c88f P 621eca29e73c4d0e819db2ec626df1d1: Deleting consensus metadata
I20260925 02:50:17.366107 4513 catalog_manager.cc:4841] TS 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451): tablet e3fe0ca14e8747a1803908b62481c88f (table test-table [id=9941c3ec3546487a826509fe8767324b]) successfully deleted
I20260925 02:50:17.366355 4515 catalog_manager.cc:4841] TS 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67:36135): tablet e3fe0ca14e8747a1803908b62481c88f (table test-table [id=9941c3ec3546487a826509fe8767324b]) successfully deleted
I20260925 02:50:17.366395 4512 catalog_manager.cc:4841] TS 1e099f19b26f401ba131cee8de480943 (127.2.172.66:39511): tablet e3fe0ca14e8747a1803908b62481c88f (table test-table [id=9941c3ec3546487a826509fe8767324b]) successfully deleted
Sep 25 02:50:17 dist-test-slave-g1xh krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304617, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:50:17 dist-test-slave-g1xh krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304617, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:17.416170 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:17.398301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.2.172.126:43669)
0925 02:50:17.399071 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:17.399076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:17.399105 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:50:17.399271 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:17.399275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:17.399372 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:17.399490 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:17.401406 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:17.402609 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:17.404970 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:17.405588 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:17.405746 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:17.410373 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:17.410379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:17.410383 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:17.410423 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:17.413306 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:17.414069 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:17.414075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:17.414077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:17.414215 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:17.414632 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:17.414637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:17.414639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:17.414922 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:17.415196 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:17.415522 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 02:50:17.415800 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":387,"threads_started":1}
I20260925 02:50:17.420998 4526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:17.421773 4526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:17.430137 4810 tablet_service.cc:1467] Processing CreateTablet for tablet 90a68b152555455ab37dc376c83dd7be (DEFAULT_TABLE table=test-table [id=0fcbc420d4174f9c9dc981f825ae8ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:17.430136 4668 tablet_service.cc:1467] Processing CreateTablet for tablet 90a68b152555455ab37dc376c83dd7be (DEFAULT_TABLE table=test-table [id=0fcbc420d4174f9c9dc981f825ae8ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:17.430313 4951 tablet_service.cc:1467] Processing CreateTablet for tablet 90a68b152555455ab37dc376c83dd7be (DEFAULT_TABLE table=test-table [id=0fcbc420d4174f9c9dc981f825ae8ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:17.430742 4810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a68b152555455ab37dc376c83dd7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:17.430737 4668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a68b152555455ab37dc376c83dd7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:17.430951 4951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a68b152555455ab37dc376c83dd7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:17.436973 5142 tablet_bootstrap.cc:492] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1: Bootstrap starting.
I20260925 02:50:17.439308 5142 tablet_bootstrap.cc:654] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:17.439580 5143 tablet_bootstrap.cc:492] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46: Bootstrap starting.
I20260925 02:50:17.440774 5145 tablet_bootstrap.cc:492] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943: Bootstrap starting.
I20260925 02:50:17.442083 5143 tablet_bootstrap.cc:654] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:17.442713 5142 tablet_bootstrap.cc:492] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1: No bootstrap required, opened a new log
I20260925 02:50:17.442914 5142 ts_tablet_manager.cc:1397] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:50:17.442940 5145 tablet_bootstrap.cc:654] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:17.444015 5142 raft_consensus.cc:348] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.444409 5142 raft_consensus.cc:374] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:17.444533 5142 raft_consensus.cc:729] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 621eca29e73c4d0e819db2ec626df1d1, State: Initialized, Role: FOLLOWER
I20260925 02:50:17.444578 5143 tablet_bootstrap.cc:492] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46: No bootstrap required, opened a new log
I20260925 02:50:17.444752 5143 ts_tablet_manager.cc:1397] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:50:17.444933 5142 consensus_queue.cc:260] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.445973 5142 ts_tablet_manager.cc:1428] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:17.446652 5143 raft_consensus.cc:348] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.446992 5143 raft_consensus.cc:374] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:17.447129 5143 raft_consensus.cc:729] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8310947b68724400bb7d8107b8b62b46, State: Initialized, Role: FOLLOWER
I20260925 02:50:17.447458 5143 consensus_queue.cc:260] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.447863 5145 tablet_bootstrap.cc:492] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943: No bootstrap required, opened a new log
I20260925 02:50:17.448062 5145 ts_tablet_manager.cc:1397] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.006s
I20260925 02:50:17.449107 5143 ts_tablet_manager.cc:1428] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 02:50:17.449326 5145 raft_consensus.cc:348] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.449586 5145 raft_consensus.cc:374] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:17.449694 5145 raft_consensus.cc:729] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e099f19b26f401ba131cee8de480943, State: Initialized, Role: FOLLOWER
I20260925 02:50:17.450016 5145 consensus_queue.cc:260] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.450726 5145 ts_tablet_manager.cc:1428] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:50:17.458844 5049 raft_consensus.cc:482] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:17.459079 5049 raft_consensus.cc:504] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.459893 5049 leader_election.cc:288] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451), 1e099f19b26f401ba131cee8de480943 (127.2.172.66:39511)
I20260925 02:50:17.460655 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a68b152555455ab37dc376c83dd7be" candidate_uuid: "621eca29e73c4d0e819db2ec626df1d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8310947b68724400bb7d8107b8b62b46" is_pre_election: true
I20260925 02:50:17.460691 4831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a68b152555455ab37dc376c83dd7be" candidate_uuid: "621eca29e73c4d0e819db2ec626df1d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e099f19b26f401ba131cee8de480943" is_pre_election: true
I20260925 02:50:17.460961 4694 raft_consensus.cc:2454] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 621eca29e73c4d0e819db2ec626df1d1 in term 0.
I20260925 02:50:17.461014 4831 raft_consensus.cc:2454] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 621eca29e73c4d0e819db2ec626df1d1 in term 0.
I20260925 02:50:17.461588 4904 leader_election.cc:302] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 621eca29e73c4d0e819db2ec626df1d1, 8310947b68724400bb7d8107b8b62b46; no voters:
I20260925 02:50:17.462054 5049 raft_consensus.cc:2781] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:17.462248 5049 raft_consensus.cc:482] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:17.462358 5049 raft_consensus.cc:3037] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:17.464818 5049 raft_consensus.cc:504] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.465541 5049 leader_election.cc:288] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [CANDIDATE]: Term 1 election: Requested vote from peers 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451), 1e099f19b26f401ba131cee8de480943 (127.2.172.66:39511)
I20260925 02:50:17.466329 4831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a68b152555455ab37dc376c83dd7be" candidate_uuid: "621eca29e73c4d0e819db2ec626df1d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e099f19b26f401ba131cee8de480943"
I20260925 02:50:17.466645 4831 raft_consensus.cc:3037] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:17.467836 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a68b152555455ab37dc376c83dd7be" candidate_uuid: "621eca29e73c4d0e819db2ec626df1d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8310947b68724400bb7d8107b8b62b46"
I20260925 02:50:17.468130 4694 raft_consensus.cc:3037] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:17.469621 4831 raft_consensus.cc:2454] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 621eca29e73c4d0e819db2ec626df1d1 in term 1.
I20260925 02:50:17.470326 4903 leader_election.cc:302] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e099f19b26f401ba131cee8de480943, 621eca29e73c4d0e819db2ec626df1d1; no voters:
I20260925 02:50:17.470808 5049 raft_consensus.cc:2781] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:17.470884 4694 raft_consensus.cc:2454] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 621eca29e73c4d0e819db2ec626df1d1 in term 1.
I20260925 02:50:17.471104 5049 raft_consensus.cc:686] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 1 LEADER]: Becoming Leader. State: Replica: 621eca29e73c4d0e819db2ec626df1d1, State: Running, Role: LEADER
I20260925 02:50:17.471506 5049 consensus_queue.cc:237] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.474596 4527 catalog_manager.cc:5495] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67). New cstate: current_term: 1 leader_uuid: "621eca29e73c4d0e819db2ec626df1d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } health_report { overall_health: HEALTHY } } }
I20260925 02:50:17.505702 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:17.499600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.2.172.67:36135)
0925 02:50:17.500225 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:17.500230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:17.500254 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:50:17.500335 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:17.500338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:17.500456 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:17.500560 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:17.501041 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:17.501862 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:17.503533 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:17.504101 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:17.504222 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:17.504738 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:17.504861 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:17.504938 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:17.505180 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 02:50:17.505261 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":356,"threads_started":1}
I20260925 02:50:17.524385 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:17.517371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:43333 (local address 127.2.172.67:36135)
0925 02:50:17.517658 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:17.517665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:17.517993 (+ 328us) server_negotiation.cc:407] Connection header received
0925 02:50:17.518176 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:17.518183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:17.518296 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:17.518440 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:17.518969 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:17.520203 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:17.521977 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:17.522719 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:17.522883 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:17.523382 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:17.523584 (+ 202us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:17.523676 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:17.523900 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 02:50:17.524003 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260925 02:50:17.524426 5155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:17.517229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:36135 (local address 127.2.172.67:43333)
0925 02:50:17.517875 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:17.517892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:17.518024 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:17.518508 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:17.518512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:17.518526 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:50:17.518775 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:17.518783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:17.520387 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:17.520391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:17.521757 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:17.521776 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:17.522968 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:17.522973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:17.523133 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:17.523182 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:50:17.523760 (+ 578us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:50:17.523770 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 02:50:17.523846 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:50:17.523968 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":326,"threads_started":1}
I20260925 02:50:17.535175 4694 raft_consensus.cc:1261] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Refusing update from remote peer 621eca29e73c4d0e819db2ec626df1d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:17.535187 4831 raft_consensus.cc:1261] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Refusing update from remote peer 621eca29e73c4d0e819db2ec626df1d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:17.535997 5049 consensus_queue.cc:1035] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:17.536379 5117 consensus_queue.cc:1035] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:17.543289 5114 mvcc.cc:205] Tried to move back new op lower bound from 7333087713418407936 to 7333087713166135296. Current Snapshot: MvccSnapshot[applied={T|T < 7333087713418407936}]
I20260925 02:50:17.603785 4526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:17.604110 4526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:17.608377 4526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe: Sending DeleteTablet for 3 replicas of tablet 90a68b152555455ab37dc376c83dd7be
I20260925 02:50:17.609450 4668 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a68b152555455ab37dc376c83dd7be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:17 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:57538
I20260925 02:50:17.609773 5133 tablet_replica.cc:330] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46: stopping tablet replica
I20260925 02:50:17.609732 4951 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a68b152555455ab37dc376c83dd7be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:17 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:43692
I20260925 02:50:17.609989 5133 raft_consensus.cc:2229] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:17.610100 5131 tablet_replica.cc:330] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1: stopping tablet replica
I20260925 02:50:17.610179 5133 raft_consensus.cc:2258] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:17.610369 5131 raft_consensus.cc:2229] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:50:17.610931 4810 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a68b152555455ab37dc376c83dd7be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:17 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:38748
I20260925 02:50:17.611078 5131 raft_consensus.cc:2258] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:17.611368 5132 tablet_replica.cc:330] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943: stopping tablet replica
I20260925 02:50:17.611656 5132 raft_consensus.cc:2229] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:17.611876 5132 raft_consensus.cc:2258] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:17.612638 5131 ts_tablet_manager.cc:1905] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:17.612920 4526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:17.613656 4526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:17.622082 5131 ts_tablet_manager.cc:1918] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:50:17.622006 4951 tablet_service.cc:1467] Processing CreateTablet for tablet df036c7197034642981c8cd68b294d4c (DEFAULT_TABLE table=test-table [id=c6539d286b6245eb83859beb24242850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:17.622632 4951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df036c7197034642981c8cd68b294d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:17.623023 4668 tablet_service.cc:1467] Processing CreateTablet for tablet df036c7197034642981c8cd68b294d4c (DEFAULT_TABLE table=test-table [id=c6539d286b6245eb83859beb24242850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:17.623529 4668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df036c7197034642981c8cd68b294d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:17.628535 4810 tablet_service.cc:1467] Processing CreateTablet for tablet df036c7197034642981c8cd68b294d4c (DEFAULT_TABLE table=test-table [id=c6539d286b6245eb83859beb24242850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:17.629037 4810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df036c7197034642981c8cd68b294d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:17.631079 5142 tablet_bootstrap.cc:492] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1: Bootstrap starting.
I20260925 02:50:17.633533 5142 tablet_bootstrap.cc:654] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:17.638876 5132 ts_tablet_manager.cc:1905] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:17.641595 5143 tablet_bootstrap.cc:492] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46: Bootstrap starting.
I20260925 02:50:17.643509 5143 tablet_bootstrap.cc:654] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:17.644485 5145 tablet_bootstrap.cc:492] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943: Bootstrap starting.
I20260925 02:50:17.645965 5133 ts_tablet_manager.cc:1905] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:17.646596 5145 tablet_bootstrap.cc:654] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:17.647109 5132 ts_tablet_manager.cc:1918] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:50:17.647272 5132 log.cc:1196] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/90a68b152555455ab37dc376c83dd7be
I20260925 02:50:17.648407 5132 ts_tablet_manager.cc:1939] T 90a68b152555455ab37dc376c83dd7be P 1e099f19b26f401ba131cee8de480943: Deleting consensus metadata
I20260925 02:50:17.650174 4512 catalog_manager.cc:4841] TS 1e099f19b26f401ba131cee8de480943 (127.2.172.66:39511): tablet 90a68b152555455ab37dc376c83dd7be (table test-table [id=0fcbc420d4174f9c9dc981f825ae8ad3]) successfully deleted
I20260925 02:50:17.650806 5131 log.cc:1196] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/90a68b152555455ab37dc376c83dd7be
I20260925 02:50:17.651401 5131 ts_tablet_manager.cc:1939] T 90a68b152555455ab37dc376c83dd7be P 621eca29e73c4d0e819db2ec626df1d1: Deleting consensus metadata
I20260925 02:50:17.653021 5142 tablet_bootstrap.cc:492] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1: No bootstrap required, opened a new log
I20260925 02:50:17.653252 5142 ts_tablet_manager.cc:1397] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20260925 02:50:17.653699 5133 ts_tablet_manager.cc:1918] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:50:17.653995 5133 log.cc:1196] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/90a68b152555455ab37dc376c83dd7be
I20260925 02:50:17.654641 5133 ts_tablet_manager.cc:1939] T 90a68b152555455ab37dc376c83dd7be P 8310947b68724400bb7d8107b8b62b46: Deleting consensus metadata
I20260925 02:50:17.654474 5142 raft_consensus.cc:348] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.654868 5142 raft_consensus.cc:374] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:17.654975 5142 raft_consensus.cc:729] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 621eca29e73c4d0e819db2ec626df1d1, State: Initialized, Role: FOLLOWER
I20260925 02:50:17.655301 5142 consensus_queue.cc:260] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.657615 5142 ts_tablet_manager.cc:1428] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 02:50:17.658885 4515 catalog_manager.cc:4841] TS 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67:36135): tablet 90a68b152555455ab37dc376c83dd7be (table test-table [id=0fcbc420d4174f9c9dc981f825ae8ad3]) successfully deleted
I20260925 02:50:17.659502 4513 catalog_manager.cc:4841] TS 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451): tablet 90a68b152555455ab37dc376c83dd7be (table test-table [id=0fcbc420d4174f9c9dc981f825ae8ad3]) successfully deleted
I20260925 02:50:17.661588 5143 tablet_bootstrap.cc:492] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46: No bootstrap required, opened a new log
I20260925 02:50:17.661844 5143 ts_tablet_manager.cc:1397] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.000s
I20260925 02:50:17.662811 5143 raft_consensus.cc:348] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.663121 5143 raft_consensus.cc:374] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:17.663213 5143 raft_consensus.cc:729] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8310947b68724400bb7d8107b8b62b46, State: Initialized, Role: FOLLOWER
I20260925 02:50:17.663605 5143 consensus_queue.cc:260] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.664268 5143 ts_tablet_manager.cc:1428] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:50:17.664563 5145 tablet_bootstrap.cc:492] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943: No bootstrap required, opened a new log
I20260925 02:50:17.664743 5145 ts_tablet_manager.cc:1397] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.002s
I20260925 02:50:17.665585 5145 raft_consensus.cc:348] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.665853 5145 raft_consensus.cc:374] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:17.665961 5145 raft_consensus.cc:729] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e099f19b26f401ba131cee8de480943, State: Initialized, Role: FOLLOWER
I20260925 02:50:17.666293 5145 consensus_queue.cc:260] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:17.666936 5145 ts_tablet_manager.cc:1428] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 02:50:18.039451 5149 raft_consensus.cc:482] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:18.039801 5149 raft_consensus.cc:504] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:18.040643 5149 leader_election.cc:288] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e099f19b26f401ba131cee8de480943 (127.2.172.66:39511), 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67:36135)
I20260925 02:50:18.041355 4976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df036c7197034642981c8cd68b294d4c" candidate_uuid: "8310947b68724400bb7d8107b8b62b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621eca29e73c4d0e819db2ec626df1d1" is_pre_election: true
I20260925 02:50:18.041414 4830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df036c7197034642981c8cd68b294d4c" candidate_uuid: "8310947b68724400bb7d8107b8b62b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e099f19b26f401ba131cee8de480943" is_pre_election: true
I20260925 02:50:18.041621 4976 raft_consensus.cc:2454] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8310947b68724400bb7d8107b8b62b46 in term 0.
I20260925 02:50:18.041793 4830 raft_consensus.cc:2454] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8310947b68724400bb7d8107b8b62b46 in term 0.
I20260925 02:50:18.042268 4622 leader_election.cc:302] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 621eca29e73c4d0e819db2ec626df1d1, 8310947b68724400bb7d8107b8b62b46; no voters:
I20260925 02:50:18.042810 5149 raft_consensus.cc:2781] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:18.042979 5149 raft_consensus.cc:482] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:18.043102 5149 raft_consensus.cc:3037] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:18.045828 5149 raft_consensus.cc:504] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:18.046846 5149 leader_election.cc:288] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [CANDIDATE]: Term 1 election: Requested vote from peers 1e099f19b26f401ba131cee8de480943 (127.2.172.66:39511), 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67:36135)
I20260925 02:50:18.047498 4830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df036c7197034642981c8cd68b294d4c" candidate_uuid: "8310947b68724400bb7d8107b8b62b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e099f19b26f401ba131cee8de480943"
I20260925 02:50:18.047538 4976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df036c7197034642981c8cd68b294d4c" candidate_uuid: "8310947b68724400bb7d8107b8b62b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621eca29e73c4d0e819db2ec626df1d1"
I20260925 02:50:18.047850 4830 raft_consensus.cc:3037] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:18.047885 4976 raft_consensus.cc:3037] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:18.051148 4830 raft_consensus.cc:2454] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8310947b68724400bb7d8107b8b62b46 in term 1.
I20260925 02:50:18.051154 4976 raft_consensus.cc:2454] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8310947b68724400bb7d8107b8b62b46 in term 1.
I20260925 02:50:18.051776 4622 leader_election.cc:302] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 621eca29e73c4d0e819db2ec626df1d1, 8310947b68724400bb7d8107b8b62b46; no voters:
I20260925 02:50:18.052165 5149 raft_consensus.cc:2781] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:18.052713 5149 raft_consensus.cc:686] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 1 LEADER]: Becoming Leader. State: Replica: 8310947b68724400bb7d8107b8b62b46, State: Running, Role: LEADER
I20260925 02:50:18.053206 5149 consensus_queue.cc:237] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } }
I20260925 02:50:18.057708 4527 catalog_manager.cc:5495] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8310947b68724400bb7d8107b8b62b46 (127.2.172.65). New cstate: current_term: 1 leader_uuid: "8310947b68724400bb7d8107b8b62b46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8310947b68724400bb7d8107b8b62b46" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 } health_report { overall_health: UNKNOWN } } }
I20260925 02:50:18.140419 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:18.132754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.2.172.65:34451)
0925 02:50:18.133451 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:18.133456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:18.133492 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:50:18.133969 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:18.133973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:18.134088 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:18.134282 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:18.134853 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:18.135715 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:18.137370 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:18.137993 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:18.138159 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:18.138726 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:18.139594 (+ 868us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:18.139771 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:18.139885 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 02:50:18.139968 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":417,"threads_started":1}
I20260925 02:50:18.159824 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:18.154060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:60681 (local address 127.2.172.67:36135)
0925 02:50:18.154607 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:18.154615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:18.154712 (+ 97us) server_negotiation.cc:407] Connection header received
0925 02:50:18.154975 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:18.154979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:18.155052 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:18.155164 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:18.155740 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:18.156484 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:18.157938 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:18.158400 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:18.158545 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:18.158984 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:18.159105 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:18.159207 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:18.159351 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:50:18.159432 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":253,"threads_started":1}
I20260925 02:50:18.159920 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:18.153711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:36135 (local address 127.2.172.65:60681)
0925 02:50:18.154576 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:18.154602 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:18.154820 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:18.155180 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:18.155184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:18.155196 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:50:18.155534 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:18.155544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:18.156629 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:18.156634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:18.157778 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:18.157790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:18.158572 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:18.158578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:18.158720 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:18.158785 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:50:18.159209 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:50:18.159223 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 02:50:18.159328 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 02:50:18.159495 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":400,"threads_started":1}
I20260925 02:50:18.170524 4830 raft_consensus.cc:1261] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Refusing update from remote peer 8310947b68724400bb7d8107b8b62b46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:18.170774 4976 raft_consensus.cc:1261] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Refusing update from remote peer 8310947b68724400bb7d8107b8b62b46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:18.171353 5164 consensus_queue.cc:1035] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e099f19b26f401ba131cee8de480943" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:18.171775 5149 consensus_queue.cc:1035] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "621eca29e73c4d0e819db2ec626df1d1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:18.201846 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:18.195987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:55389 (local address 127.2.172.65:34451)
0925 02:50:18.196295 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:18.196303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:18.196705 (+ 402us) server_negotiation.cc:407] Connection header received
0925 02:50:18.196915 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:18.196921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:18.197020 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:18.197133 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:18.197620 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:18.198402 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:18.199827 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:18.200539 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:18.200701 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:18.201194 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:18.201328 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:18.201405 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:18.201511 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 02:50:18.201587 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260925 02:50:18.201880 5185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:18.195868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:34451 (local address 127.2.172.67:55389)
0925 02:50:18.196584 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:18.196607 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:18.196760 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:18.197151 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:18.197157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:18.197169 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:50:18.197456 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:18.197465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:18.198548 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:18.198555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:18.199681 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:18.199693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:18.200696 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:18.200704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:18.200924 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:18.200980 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:50:18.201422 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:50:18.201431 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:50:18.201495 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 02:50:18.201610 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":368,"threads_started":1}
W20260925 02:50:18.203174 5169 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:50:19.815327 4526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:19.815675 4526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:19.820200 4526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6f3d33331a340429be35f7dbc86bcfe: Sending DeleteTablet for 3 replicas of tablet df036c7197034642981c8cd68b294d4c
I20260925 02:50:19.821168 4668 tablet_service.cc:1514] Processing DeleteTablet for tablet df036c7197034642981c8cd68b294d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:19 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:57538
I20260925 02:50:19.821259 4951 tablet_service.cc:1514] Processing DeleteTablet for tablet df036c7197034642981c8cd68b294d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:19 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:43692
I20260925 02:50:19.821321 4810 tablet_service.cc:1514] Processing DeleteTablet for tablet df036c7197034642981c8cd68b294d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:19 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:38748
I20260925 02:50:19.822084 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 4602
I20260925 02:50:19.837308 5212 tablet_replica.cc:330] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943: stopping tablet replica
I20260925 02:50:19.837649 5212 raft_consensus.cc:2229] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:19.837993 5212 raft_consensus.cc:2258] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:19.839816 5212 ts_tablet_manager.cc:1905] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:19.860121 5210 tablet_replica.cc:330] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46: stopping tablet replica
I20260925 02:50:19.860437 5210 raft_consensus.cc:2229] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:50:19.861166 5212 ts_tablet_manager.cc:1918] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:50:19.861938 5210 raft_consensus.cc:2258] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:19.861982 5212 log.cc:1196] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/df036c7197034642981c8cd68b294d4c
I20260925 02:50:19.862701 5212 ts_tablet_manager.cc:1939] T df036c7197034642981c8cd68b294d4c P 1e099f19b26f401ba131cee8de480943: Deleting consensus metadata
I20260925 02:50:19.863334 5210 ts_tablet_manager.cc:1905] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:19.864269 4512 catalog_manager.cc:4841] TS 1e099f19b26f401ba131cee8de480943 (127.2.172.66:39511): tablet df036c7197034642981c8cd68b294d4c (table test-table [id=c6539d286b6245eb83859beb24242850]) successfully deleted
I20260925 02:50:19.865576 5211 tablet_replica.cc:330] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1: stopping tablet replica
Sep 25 02:50:19 dist-test-slave-g1xh krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304617, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:19.865824 5211 raft_consensus.cc:2229] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:19.866199 5211 raft_consensus.cc:2258] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:19.867594 5211 ts_tablet_manager.cc:1905] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:19.872713 5210 ts_tablet_manager.cc:1918] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:50:19.872888 5210 log.cc:1196] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/df036c7197034642981c8cd68b294d4c
I20260925 02:50:19.873189 5211 ts_tablet_manager.cc:1918] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:50:19.873373 5211 log.cc:1196] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/df036c7197034642981c8cd68b294d4c
I20260925 02:50:19.873492 5210 ts_tablet_manager.cc:1939] T df036c7197034642981c8cd68b294d4c P 8310947b68724400bb7d8107b8b62b46: Deleting consensus metadata
I20260925 02:50:19.873893 5211 ts_tablet_manager.cc:1939] T df036c7197034642981c8cd68b294d4c P 621eca29e73c4d0e819db2ec626df1d1: Deleting consensus metadata
I20260925 02:50:19.874974 4513 catalog_manager.cc:4841] TS 8310947b68724400bb7d8107b8b62b46 (127.2.172.65:34451): tablet df036c7197034642981c8cd68b294d4c (table test-table [id=c6539d286b6245eb83859beb24242850]) successfully deleted
I20260925 02:50:19.875419 4515 catalog_manager.cc:4841] TS 621eca29e73c4d0e819db2ec626df1d1 (127.2.172.67:36135): tablet df036c7197034642981c8cd68b294d4c (table test-table [id=c6539d286b6245eb83859beb24242850]) successfully deleted
I20260925 02:50:19.875919 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:19.823881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.2.172.65:34451)
0925 02:50:19.852454 (+ 28573us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:19.852461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:19.852491 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:50:19.852571 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:19.852576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:19.852667 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:19.852809 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:19.855063 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:19.856434 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:19.858387 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:19.859154 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:19.859284 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:19.867311 (+ 8027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:19.867334 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:19.867362 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:19.867418 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:19.872006 (+ 4588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:19.873070 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:19.873086 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:19.873093 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:19.873313 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:19.873862 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:19.873867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:19.873870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:19.874397 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:19.874754 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:19.875239 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 02:50:19.875497 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28389,"thread_start_us":35591,"threads_started":1}
W20260925 02:50:19.876703 4736 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57556
W20260925 02:50:19.877435 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 4602: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:50:19.877704 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 4602
I20260925 02:50:19.908756 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 4745
Sep 25 02:50:19 dist-test-slave-g1xh krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304617, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:50:19.926596 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:19.909742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.2.172.66:39511)
0925 02:50:19.910285 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:19.910290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:19.910308 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:50:19.910375 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:19.910380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:19.910454 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:19.910558 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:19.912032 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:19.912886 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:19.914324 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:19.914727 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:19.914827 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:19.919915 (+ 5088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:19.919937 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:19.919970 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:19.920012 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:19.923411 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:19.924251 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:19.924260 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:19.924263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:19.924406 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:19.924846 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:19.924850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:19.924852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:19.925311 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:19.925583 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:19.925929 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 02:50:19.926139 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":288,"threads_started":1}
W20260925 02:50:19.927493 4877 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38776
W20260925 02:50:19.928483 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 4745: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:50:19.928642 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 4745
I20260925 02:50:19.957859 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 4886
Sep 25 02:50:19 dist-test-slave-g1xh krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304617, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:50:19.976617 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:19.958728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.2.172.67:36135)
0925 02:50:19.959335 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:19.959340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:19.959358 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:50:19.959424 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:19.959428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:19.959495 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:19.959681 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:19.961133 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:19.962012 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:19.963503 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:19.964166 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:19.964289 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:19.969294 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:19.969313 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:19.969336 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:19.969387 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:19.972713 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:19.973516 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:19.973601 (+ 85us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:19.973604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:19.973885 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:19.974440 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:19.974446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:19.974450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:19.975084 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:19.975438 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:19.975872 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 02:50:19.976164 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":335,"threads_started":1}
W20260925 02:50:19.977439 5018 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43730
W20260925 02:50:19.978209 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 4886: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:50:19.978434 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 4886
I20260925 02:50:20.011330 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 4495
I20260925 02:50:20.013011 4570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:20.152266 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 4495
2026-09-25T02:50:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9172 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:20 dist-test-slave-g1xh krb5kdc[5220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:20 dist-test-slave-g1xh krb5kdc[5220](info): set up 2 sockets
Sep 25 02:50:20 dist-test-slave-g1xh krb5kdc[5220](info): commencing operation
krb5kdc: starting...
W20260925 02:50:21.283429 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 02:50:21.504487 2737 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.151s user 0.000s sys 0.008s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/test-user.keytab.
W20260925 02:50:22.229897 2737 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.725s user 0.000s sys 0.007s
Sep 25 02:50:22 dist-test-slave-g1xh krb5kdc[5220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
W20260925 02:50:23.262835 2737 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.172.126: real 0.956s user 0.000s sys 0.016s
I20260925 02:50:23.264182 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:46565
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:46565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:23.669111 5234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:23.669603 5234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:23.669778 5234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:23.678725 5234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:23.678944 5234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:23.679065 5234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:23.679178 5234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:23.679260 5234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:23.691027 5234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:46565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:23.693063 5234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:23.694873 5234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:23.710423 5239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:23.710490 5240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:23.710505 5242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:23.711290 5234 server_base.cc:1034] running on GCE node
I20260925 02:50:23.712467 5234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:23.714373 5234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:23.715716 5234 hybrid_clock.cc:648] HybridClock initialized: now 1790304623715653 us; error 51 us; skew 500 ppm
I20260925 02:50:23.716200 5234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:23 dist-test-slave-g1xh krb5kdc[5220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304623, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:23.727023 5234 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:50:23.729351 5234 webserver.cc:460] Webserver started at http://127.2.172.126:40261/ using document root <none> and password file <none>
I20260925 02:50:23.730350 5234 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:23.730516 5234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:23.730974 5234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:23.734989 5234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "0fd547aa366f440fb7da928714f7de20"
format_stamp: "Formatted at 2026-09-25 02:50:23 on dist-test-slave-g1xh"
server_key: "5c2f6a79ce694e61fb6fcf63fbf83c27"
server_key_iv: "c3abe9c95afa4104fcc4c98d79f08759"
server_key_version: "encryptionkey@0"
I20260925 02:50:23.736028 5234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "0fd547aa366f440fb7da928714f7de20"
format_stamp: "Formatted at 2026-09-25 02:50:23 on dist-test-slave-g1xh"
server_key: "5c2f6a79ce694e61fb6fcf63fbf83c27"
server_key_iv: "c3abe9c95afa4104fcc4c98d79f08759"
server_key_version: "encryptionkey@0"
I20260925 02:50:23.742575 5234 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260925 02:50:23.746984 5249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:23.748945 5234 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:50:23.749190 5234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "0fd547aa366f440fb7da928714f7de20"
format_stamp: "Formatted at 2026-09-25 02:50:23 on dist-test-slave-g1xh"
server_key: "5c2f6a79ce694e61fb6fcf63fbf83c27"
server_key_iv: "c3abe9c95afa4104fcc4c98d79f08759"
server_key_version: "encryptionkey@0"
I20260925 02:50:23.749444 5234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:23.769701 5234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:23.774849 5234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:23.775188 5234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:23.797015 5234 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:46565
I20260925 02:50:23.797021 5302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:46565 every 8 connection(s)
I20260925 02:50:23.798955 5234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:23.803097 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 5234
I20260925 02:50:23.803377 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:23.804368 2737 external_mini_cluster.cc:1428] Setting key 76054053e443644bd145e549d1d2160d
I20260925 02:50:23.805442 5303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:50:23 dist-test-slave-g1xh krb5kdc[5220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:23.823693 5303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20: Bootstrap starting.
I20260925 02:50:23.828274 5303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:23.829919 5303 log.cc:824] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:23.833042 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:23.807028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.2.172.126:46565)
0925 02:50:23.808048 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:23.808065 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:23.808139 (+ 74us) server_negotiation.cc:407] Connection header received
0925 02:50:23.810161 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:23.810193 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:23.810909 (+ 716us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:23.811382 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:23.813599 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:23.815236 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:23.817045 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:23.817881 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:23.818035 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:23.823775 (+ 5740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:23.823785 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:23.823807 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:23.823891 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:23.827423 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:23.828518 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:23.828534 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:23.828546 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:23.828846 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:23.829431 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:23.829436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:23.829438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:23.830171 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:23.830444 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:23.831124 (+ 680us) server_negotiation.cc:299] Negotiation successful
0925 02:50:23.831742 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":489,"threads_started":1}
I20260925 02:50:23.833830 5303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20: No bootstrap required, opened a new log
I20260925 02:50:23.842269 5303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd547aa366f440fb7da928714f7de20" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46565 } }
I20260925 02:50:23.842896 5303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:23.843044 5303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fd547aa366f440fb7da928714f7de20, State: Initialized, Role: FOLLOWER
I20260925 02:50:23.843856 5303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd547aa366f440fb7da928714f7de20" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46565 } }
I20260925 02:50:23.844146 5303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:23.844329 5303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:23.844537 5303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:23.847261 5303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd547aa366f440fb7da928714f7de20" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46565 } }
I20260925 02:50:23.848132 5303 leader_election.cc:302] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fd547aa366f440fb7da928714f7de20; no voters:
I20260925 02:50:23.849054 5303 leader_election.cc:288] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:23.849237 5308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:23.850208 5308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [term 1 LEADER]: Becoming Leader. State: Replica: 0fd547aa366f440fb7da928714f7de20, State: Running, Role: LEADER
I20260925 02:50:23.850890 5308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd547aa366f440fb7da928714f7de20" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46565 } }
I20260925 02:50:23.851971 5303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:23.855504 5310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fd547aa366f440fb7da928714f7de20. Latest consensus state: current_term: 1 leader_uuid: "0fd547aa366f440fb7da928714f7de20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd547aa366f440fb7da928714f7de20" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46565 } } }
I20260925 02:50:23.856055 5310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:23.856899 5309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fd547aa366f440fb7da928714f7de20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd547aa366f440fb7da928714f7de20" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46565 } } }
I20260925 02:50:23.857336 5309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:23.868878 5317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:23.875449 5317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:23.887382 5317 catalog_manager.cc:1347] Generated new cluster ID: 9aa8938e9cac4552baacf9e192c1ac27
I20260925 02:50:23.887590 5317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:23.921176 5317 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:23.923523 5317 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:23.935343 5317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fd547aa366f440fb7da928714f7de20: Generated new TSK 0
I20260925 02:50:23.936913 5317 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:50:24.000138 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:46565
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:24.410724 5329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:24.411151 5329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:24.411365 5329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:24.420109 5329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:24.420338 5329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:24.420557 5329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:50:24.432029 5329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:46565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:24.433713 5329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:24.436069 5329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:24.452577 5334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:24.452769 5337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:24.452593 5335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:24.454149 5329 server_base.cc:1034] running on GCE node
I20260925 02:50:24.455025 5329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:24.456466 5329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:24.457729 5329 hybrid_clock.cc:648] HybridClock initialized: now 1790304624457683 us; error 42 us; skew 500 ppm
I20260925 02:50:24.458134 5329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:24 dist-test-slave-g1xh krb5kdc[5220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:24.462284 5329 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:50:24.464289 5329 webserver.cc:460] Webserver started at http://127.2.172.65:46789/ using document root <none> and password file <none>
I20260925 02:50:24.465183 5329 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:24.465353 5329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:24.465718 5329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:24.469280 5329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "8a71292711764dd6ac63a10b1ae66f91"
format_stamp: "Formatted at 2026-09-25 02:50:24 on dist-test-slave-g1xh"
server_key: "48e86291f57e8871c5c13d2d86e562b5"
server_key_iv: "a057160a88e4a1fec160796403e07577"
server_key_version: "encryptionkey@0"
I20260925 02:50:24.470237 5329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "8a71292711764dd6ac63a10b1ae66f91"
format_stamp: "Formatted at 2026-09-25 02:50:24 on dist-test-slave-g1xh"
server_key: "48e86291f57e8871c5c13d2d86e562b5"
server_key_iv: "a057160a88e4a1fec160796403e07577"
server_key_version: "encryptionkey@0"
I20260925 02:50:24.476369 5329 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260925 02:50:24.480669 5344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:24.482534 5329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:50:24.482782 5329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "8a71292711764dd6ac63a10b1ae66f91"
format_stamp: "Formatted at 2026-09-25 02:50:24 on dist-test-slave-g1xh"
server_key: "48e86291f57e8871c5c13d2d86e562b5"
server_key_iv: "a057160a88e4a1fec160796403e07577"
server_key_version: "encryptionkey@0"
I20260925 02:50:24.483060 5329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:24.505234 5329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:24.510535 5329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:24.510881 5329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:24.512351 5329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:24.513942 5329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:24.514055 5329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:24.514222 5329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:24.514298 5329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:24.551198 5329 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:36875
I20260925 02:50:24.551249 5459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:36875 every 8 connection(s)
I20260925 02:50:24.553066 5329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:24.557780 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 5329
I20260925 02:50:24.558044 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:24.558795 2737 external_mini_cluster.cc:1428] Setting key 62c248bbdf54a25befeb1707accf489f
Sep 25 02:50:24 dist-test-slave-g1xh krb5kdc[5220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:24.576043 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:24.556277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:47011 (local address 127.2.172.126:46565)
0925 02:50:24.556762 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:24.556769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:24.558004 (+ 1235us) server_negotiation.cc:407] Connection header received
0925 02:50:24.560203 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:24.560210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:24.560320 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:24.560454 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:24.562297 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:24.563092 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:24.564829 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:24.565277 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:24.565469 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:24.569826 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:24.569832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:24.569835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:24.569872 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:24.572614 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:24.573515 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:24.573520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:24.573523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:24.573648 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:24.574128 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:24.574133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:24.574135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:24.574418 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:24.574591 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:24.575370 (+ 779us) server_negotiation.cc:299] Negotiation successful
0925 02:50:24.575664 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":219,"threads_started":1}
I20260925 02:50:24.577140 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:24.556975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:46565 (local address 127.2.172.65:47011)
0925 02:50:24.557760 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:24.557806 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:24.559877 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:24.560857 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:24.560873 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:24.561452 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:24.562106 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:24.562127 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:24.563273 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:24.563277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:24.564664 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:24.564678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:24.565562 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:24.565571 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:24.565825 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:24.566549 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:24.566593 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:24.569572 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:24.572780 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:24.572791 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:24.572811 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:24.573316 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:24.573770 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:24.573774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:24.573778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:24.573992 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:24.574570 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:24.574579 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:24.574965 (+ 386us) client_negotiation.cc:769] Sending connection context
0925 02:50:24.575252 (+ 287us) client_negotiation.cc:240] Negotiation successful
0925 02:50:24.575881 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":374,"threads_started":1}
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
I20260925 02:50:24.578972 5460 heartbeater.cc:347] Connected to a master server at 127.2.172.126:46565
I20260925 02:50:24.579335 5460 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:24.580312 5460 heartbeater.cc:511] Master 127.2.172.126:46565 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
I20260925 02:50:24.583029 5266 ts_manager.cc:194] Registered new tserver with Master: 8a71292711764dd6ac63a10b1ae66f91 (127.2.172.65:36875)
I20260925 02:50:24.584784 5266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:47011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:50:24.609097 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:46565
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:24.974153 5466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:24.974454 5466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:24.974550 5466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:24.981876 5466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:24.982005 5466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:24.982146 5466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:50:24.992472 5466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:46565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:24.993788 5466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:24.995082 5466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:25.008760 5471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:25.009065 5472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:25.009145 5474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:25.010422 5466 server_base.cc:1034] running on GCE node
I20260925 02:50:25.011128 5466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:25.012243 5466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:25.013480 5466 hybrid_clock.cc:648] HybridClock initialized: now 1790304625013422 us; error 46 us; skew 500 ppm
I20260925 02:50:25.013846 5466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:25 dist-test-slave-g1xh krb5kdc[5220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:25.017403 5466 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:50:25.019042 5466 webserver.cc:460] Webserver started at http://127.2.172.66:34267/ using document root <none> and password file <none>
I20260925 02:50:25.019904 5466 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:25.020066 5466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:25.020430 5466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:25.023627 5466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "9b73206349d74f039d728a99636a2663"
format_stamp: "Formatted at 2026-09-25 02:50:25 on dist-test-slave-g1xh"
server_key: "200a37fee8394bce0e119e7b0fc57c98"
server_key_iv: "6a098201b9f8c5c5da0d2fbd42cb3027"
server_key_version: "encryptionkey@0"
I20260925 02:50:25.024518 5466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "9b73206349d74f039d728a99636a2663"
format_stamp: "Formatted at 2026-09-25 02:50:25 on dist-test-slave-g1xh"
server_key: "200a37fee8394bce0e119e7b0fc57c98"
server_key_iv: "6a098201b9f8c5c5da0d2fbd42cb3027"
server_key_version: "encryptionkey@0"
I20260925 02:50:25.030124 5466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:50:25.033799 5481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.035277 5466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:50:25.035583 5466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "9b73206349d74f039d728a99636a2663"
format_stamp: "Formatted at 2026-09-25 02:50:25 on dist-test-slave-g1xh"
server_key: "200a37fee8394bce0e119e7b0fc57c98"
server_key_iv: "6a098201b9f8c5c5da0d2fbd42cb3027"
server_key_version: "encryptionkey@0"
I20260925 02:50:25.035810 5466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:25.073434 5466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:25.078083 5466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:25.078433 5466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:25.079463 5466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:25.080886 5466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:25.081001 5466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.081159 5466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:25.081236 5466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.116961 5466 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:33999
I20260925 02:50:25.117053 5596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:33999 every 8 connection(s)
I20260925 02:50:25.118336 5466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:25.121059 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 5466
I20260925 02:50:25.121243 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:25.122066 2737 external_mini_cluster.cc:1428] Setting key 0a201dd4c21361e4243bb45125ef56b2
Sep 25 02:50:25 dist-test-slave-g1xh krb5kdc[5220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:25.143811 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:25.121214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:47273 (local address 127.2.172.126:46565)
0925 02:50:25.123462 (+ 2248us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:25.123468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:25.123486 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:50:25.125183 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:25.125188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:25.125257 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:25.125363 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:25.127702 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.128451 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.131044 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.131873 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.132018 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:25.136077 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:25.136087 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:25.136094 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:25.136145 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:25.140050 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:25.141041 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:25.141050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:25.141056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:25.141229 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:25.141681 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:25.141688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:25.141695 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:25.142128 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:25.142308 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:25.143055 (+ 747us) server_negotiation.cc:299] Negotiation successful
0925 02:50:25.143312 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2118,"thread_start_us":2006,"threads_started":1}
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
I20260925 02:50:25.144629 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:25.121824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:46565 (local address 127.2.172.66:47273)
0925 02:50:25.122787 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:25.122847 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:25.124843 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:25.125666 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:25.125680 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:25.126544 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:25.127456 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:25.127487 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.128627 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.128635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:25.130865 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:25.130885 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.132145 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.132151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:25.132386 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:25.133137 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:25.133162 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:25.135799 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:25.140230 (+ 4431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:25.140241 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:25.140259 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:25.140853 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:25.141357 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:25.141360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:25.141362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:25.141549 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:25.142275 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:25.142281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:25.142638 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 02:50:25.142880 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 02:50:25.143303 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":478,"threads_started":1}
I20260925 02:50:25.146356 5597 heartbeater.cc:347] Connected to a master server at 127.2.172.126:46565
I20260925 02:50:25.146708 5597 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:25.147608 5597 heartbeater.cc:511] Master 127.2.172.126:46565 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
I20260925 02:50:25.149426 5266 ts_manager.cc:194] Registered new tserver with Master: 9b73206349d74f039d728a99636a2663 (127.2.172.66:33999)
I20260925 02:50:25.151043 5266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:47273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:50:25.175980 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:46565
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:25.563395 5603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:25.563958 5603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:25.564137 5603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:25.572767 5603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:25.572973 5603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:25.573174 5603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:50:25.585286 5603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:46565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:25.587235 5603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:25.588923 5460 heartbeater.cc:503] Master 127.2.172.126:46565 was elected leader, sending a full tablet report...
I20260925 02:50:25.589282 5603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:25.605829 5608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:25.606040 5609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:25.605860 5611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:25.607285 5603 server_base.cc:1034] running on GCE node
I20260925 02:50:25.608146 5603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:25.609386 5603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:25.610667 5603 hybrid_clock.cc:648] HybridClock initialized: now 1790304625610617 us; error 43 us; skew 500 ppm
I20260925 02:50:25.611114 5603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:25 dist-test-slave-g1xh krb5kdc[5220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:25.615592 5603 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:50:25.617616 5603 webserver.cc:460] Webserver started at http://127.2.172.67:33941/ using document root <none> and password file <none>
I20260925 02:50:25.618520 5603 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:25.618637 5603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:25.618953 5603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:25.622681 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "a398531bdc7e48249d5b886735ef92bb"
format_stamp: "Formatted at 2026-09-25 02:50:25 on dist-test-slave-g1xh"
server_key: "5cc8f412d230e2fdac0cda50af3a548f"
server_key_iv: "21dfd93c42d02f95089e126715473ea8"
server_key_version: "encryptionkey@0"
I20260925 02:50:25.623615 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "a398531bdc7e48249d5b886735ef92bb"
format_stamp: "Formatted at 2026-09-25 02:50:25 on dist-test-slave-g1xh"
server_key: "5cc8f412d230e2fdac0cda50af3a548f"
server_key_iv: "21dfd93c42d02f95089e126715473ea8"
server_key_version: "encryptionkey@0"
I20260925 02:50:25.630107 5603 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:50:25.634799 5618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.636924 5603 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 02:50:25.637130 5603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "a398531bdc7e48249d5b886735ef92bb"
format_stamp: "Formatted at 2026-09-25 02:50:25 on dist-test-slave-g1xh"
server_key: "5cc8f412d230e2fdac0cda50af3a548f"
server_key_iv: "21dfd93c42d02f95089e126715473ea8"
server_key_version: "encryptionkey@0"
I20260925 02:50:25.637367 5603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:25.663688 5603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:25.669930 5603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:25.670261 5603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:25.671533 5603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:25.673347 5603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:25.673442 5603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.673542 5603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:25.673596 5603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.713600 5603 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:40113
I20260925 02:50:25.713639 5732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:40113 every 8 connection(s)
I20260925 02:50:25.715401 5603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:25.721028 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 5603
I20260925 02:50:25.721280 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:25.721992 2737 external_mini_cluster.cc:1428] Setting key 76e2de38f81ac8d78626f07a85107ea5
Sep 25 02:50:25 dist-test-slave-g1xh krb5kdc[5220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:25.743228 5735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:25.719532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:57517 (local address 127.2.172.126:46565)
0925 02:50:25.720411 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:25.720420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:25.722172 (+ 1752us) server_negotiation.cc:407] Connection header received
0925 02:50:25.725302 (+ 3130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:25.725309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:25.725443 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:25.725588 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:25.727783 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.728668 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.730486 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.730988 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.731173 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:25.735452 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:25.735458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:25.735461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:25.735507 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:25.739035 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:25.740302 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:25.740309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:25.740311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:25.740466 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:25.740991 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:25.740996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:25.740998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:25.741346 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:25.741487 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:25.742436 (+ 949us) server_negotiation.cc:299] Negotiation successful
0925 02:50:25.742868 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":389,"threads_started":1}
I20260925 02:50:25.744627 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:25.720309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:46565 (local address 127.2.172.67:57517)
0925 02:50:25.721851 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:25.721927 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:25.724959 (+ 3032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:25.725946 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:25.725962 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:25.726730 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:25.727505 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:25.727531 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.728856 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.728862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:25.730313 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:25.730326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.731286 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.731294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:25.731660 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:25.732583 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:25.732612 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:25.735192 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:25.739228 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:25.739237 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:25.739257 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:25.740114 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:25.740593 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:25.740598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:25.740602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:25.740842 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:25.741521 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:25.741534 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:25.742008 (+ 474us) client_negotiation.cc:769] Sending connection context
0925 02:50:25.742307 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 02:50:25.742968 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"thread_start_us":507,"threads_started":1}
I20260925 02:50:25.746423 5733 heartbeater.cc:347] Connected to a master server at 127.2.172.126:46565
I20260925 02:50:25.746896 5733 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:25.747968 5733 heartbeater.cc:511] Master 127.2.172.126:46565 requested a full tablet report, sending...
I20260925 02:50:25.750180 5266 ts_manager.cc:194] Registered new tserver with Master: a398531bdc7e48249d5b886735ef92bb (127.2.172.67:40113)
I20260925 02:50:25.751456 5266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:57517
I20260925 02:50:25.752692 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:50:25.799083 5735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:50:25.795222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.2.172.126:46565)
0925 02:50:25.795567 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:25.795575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:25.796184 (+ 609us) server_negotiation.cc:407] Connection header received
0925 02:50:25.796454 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:25.796459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:25.796591 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:25.796712 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:25.798759 (+ 2047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54064: BlockingRecv error: recv got EOF from 127.0.0.1:54064 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 02:50:25.803026 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 5329
W20260925 02:50:25.807281 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 5329: Not authorized: Client connection negotiation failed: client connection to 127.2.172.65:36875: server requires authentication, but client does not have Kerberos credentials available
I20260925 02:50:25.807586 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 5329
I20260925 02:50:25.829619 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 5466
W20260925 02:50:25.833206 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 5466: Not authorized: Client connection negotiation failed: client connection to 127.2.172.66:33999: server requires authentication, but client does not have Kerberos credentials available
I20260925 02:50:25.833408 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 5466
I20260925 02:50:25.854658 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 5603
W20260925 02:50:25.858531 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 5603: Not authorized: Client connection negotiation failed: client connection to 127.2.172.67:40113: server requires authentication, but client does not have Kerberos credentials available
I20260925 02:50:25.858762 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 5603
I20260925 02:50:25.877305 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 5234
I20260925 02:50:25.878947 5298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:26.001216 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 5234
2026-09-25T02:50:26Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5844 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:26 dist-test-slave-g1xh krb5kdc[5751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:26 dist-test-slave-g1xh krb5kdc[5751](info): set up 2 sockets
Sep 25 02:50:26 dist-test-slave-g1xh krb5kdc[5751](info): commencing operation
krb5kdc: starting...
W20260925 02:50:27.155231 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:50:27 dist-test-slave-g1xh krb5kdc[5751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:50:27.372354 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:34137
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:34137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:27.720304 5764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:27.720697 5764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:27.720868 5764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:27.728297 5764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:27.728494 5764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:27.728611 5764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:27.728722 5764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:27.728817 5764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:27.739063 5764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:34137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:34137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:27.740912 5764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:27.742334 5764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:27.755038 5770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:27.755121 5769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:27.755121 5772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:27.756268 5764 server_base.cc:1034] running on GCE node
I20260925 02:50:27.757165 5764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:27.758730 5764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:27.759979 5764 hybrid_clock.cc:648] HybridClock initialized: now 1790304627759927 us; error 47 us; skew 500 ppm
I20260925 02:50:27.760399 5764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:27 dist-test-slave-g1xh krb5kdc[5751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304627, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:27.763931 5764 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:50:27.765393 5764 webserver.cc:460] Webserver started at http://127.2.172.126:43259/ using document root <none> and password file <none>
I20260925 02:50:27.766095 5764 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:27.766197 5764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:27.766461 5764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:27.769713 5764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "f796802fa0ff4609bb74b5f1e898ae20"
format_stamp: "Formatted at 2026-09-25 02:50:27 on dist-test-slave-g1xh"
server_key: "db6172e332a2cbbec6630b77a2a2c735"
server_key_iv: "a86332275e20231409f84f1c9aa7e8da"
server_key_version: "encryptionkey@0"
I20260925 02:50:27.770547 5764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "f796802fa0ff4609bb74b5f1e898ae20"
format_stamp: "Formatted at 2026-09-25 02:50:27 on dist-test-slave-g1xh"
server_key: "db6172e332a2cbbec6630b77a2a2c735"
server_key_iv: "a86332275e20231409f84f1c9aa7e8da"
server_key_version: "encryptionkey@0"
I20260925 02:50:27.776296 5764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 02:50:27.779831 5779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:27.781248 5764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:50:27.781443 5764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "f796802fa0ff4609bb74b5f1e898ae20"
format_stamp: "Formatted at 2026-09-25 02:50:27 on dist-test-slave-g1xh"
server_key: "db6172e332a2cbbec6630b77a2a2c735"
server_key_iv: "a86332275e20231409f84f1c9aa7e8da"
server_key_version: "encryptionkey@0"
I20260925 02:50:27.781642 5764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:27.819073 5764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:27.823719 5764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:27.824025 5764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:27.842801 5764 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:34137
I20260925 02:50:27.842839 5830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:34137 every 8 connection(s)
I20260925 02:50:27.844259 5764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:27.848094 5831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:27.854045 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 5764
I20260925 02:50:27.854311 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:27.854945 2737 external_mini_cluster.cc:1428] Setting key f14b58c91888e194ec49215d8888ed1f
I20260925 02:50:27.859529 5831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20: Bootstrap starting.
I20260925 02:50:27.863598 5831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:27.864893 5831 log.cc:824] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:27.868667 5831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20: No bootstrap required, opened a new log
Sep 25 02:50:27 dist-test-slave-g1xh krb5kdc[5751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:27.877218 5831 raft_consensus.cc:348] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f796802fa0ff4609bb74b5f1e898ae20" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34137 } }
I20260925 02:50:27.877661 5831 raft_consensus.cc:374] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:27.877817 5831 raft_consensus.cc:729] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f796802fa0ff4609bb74b5f1e898ae20, State: Initialized, Role: FOLLOWER
I20260925 02:50:27.878679 5831 consensus_queue.cc:260] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f796802fa0ff4609bb74b5f1e898ae20" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34137 } }
I20260925 02:50:27.878978 5831 raft_consensus.cc:388] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:27.879161 5831 raft_consensus.cc:482] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:27.879366 5831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:27.880682 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.856991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.2.172.126:34137)
0925 02:50:27.857884 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:27.857896 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:27.857939 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:50:27.859763 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:27.859794 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:27.860213 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:27.860566 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:27.862595 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.864206 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.866120 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.867003 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.867155 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.872185 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:27.872198 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:27.872216 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:27.872260 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:27.875702 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:27.876544 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:27.876553 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:27.876561 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:27.876742 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:27.877155 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:27.877161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:27.877164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:27.877897 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:27.878215 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:27.878713 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 02:50:27.879438 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":430,"threads_started":1}
I20260925 02:50:27.882524 5831 raft_consensus.cc:504] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f796802fa0ff4609bb74b5f1e898ae20" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34137 } }
I20260925 02:50:27.883109 5831 leader_election.cc:302] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f796802fa0ff4609bb74b5f1e898ae20; no voters:
I20260925 02:50:27.883906 5831 leader_election.cc:288] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:27.884058 5836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:27.885602 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:50:27.885565 5836 raft_consensus.cc:686] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [term 1 LEADER]: Becoming Leader. State: Replica: f796802fa0ff4609bb74b5f1e898ae20, State: Running, Role: LEADER
I20260925 02:50:27.886080 5836 consensus_queue.cc:237] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f796802fa0ff4609bb74b5f1e898ae20" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34137 } }
I20260925 02:50:27.886504 5831 sys_catalog.cc:564] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:27.888821 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f796802fa0ff4609bb74b5f1e898ae20. Latest consensus state: current_term: 1 leader_uuid: "f796802fa0ff4609bb74b5f1e898ae20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f796802fa0ff4609bb74b5f1e898ae20" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34137 } } }
I20260925 02:50:27.889230 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:27.889784 5837 sys_catalog.cc:455] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f796802fa0ff4609bb74b5f1e898ae20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f796802fa0ff4609bb74b5f1e898ae20" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34137 } } }
I20260925 02:50:27.890151 5837 sys_catalog.cc:458] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:27.912973 5841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:27.920639 5841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:27.942013 5841 catalog_manager.cc:1347] Generated new cluster ID: 0b68a517acfb40d39adaa4810cda1ef5
I20260925 02:50:27.942205 5841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:27.959733 5841 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:27.961809 5841 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:27.972273 5862 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:50:27.980737 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.967563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.2.172.126:34137)
0925 02:50:27.967841 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:27.967850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:27.973549 (+ 5699us) server_negotiation.cc:407] Connection header received
0925 02:50:27.973772 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:27.973779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:27.973855 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:27.973980 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:27.975093 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.975976 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.977285 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.977748 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.977902 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.979085 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:27.979090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:27.979093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:27.979874 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:27.979948 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:27.979972 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:27.980097 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:27.980241 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:50:27.980371 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 02:50:27.998939 5841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f796802fa0ff4609bb74b5f1e898ae20: Generated new TSK 0
I20260925 02:50:27.999975 5841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:50:28.010753 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 5764
I20260925 02:50:28.012107 5826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:28.106444 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 5764
2026-09-25T02:50:28Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2095 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:28 dist-test-slave-g1xh krb5kdc[5868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:28 dist-test-slave-g1xh krb5kdc[5868](info): set up 2 sockets
Sep 25 02:50:28 dist-test-slave-g1xh krb5kdc[5868](info): commencing operation
krb5kdc: starting...
W20260925 02:50:29.202203 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:50:29 dist-test-slave-g1xh krb5kdc[5868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:50:29.424726 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:35695
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:35695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:29.836472 5882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:29.836896 5882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:29.837075 5882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:29.845180 5882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:29.845314 5882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:29.845407 5882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:29.845513 5882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:29.845598 5882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:29.857059 5882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:35695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:35695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:29.859084 5882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:29.860934 5882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:29.875777 5887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:29.875643 5890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:29.875643 5888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:29.876799 5882 server_base.cc:1034] running on GCE node
I20260925 02:50:29.877838 5882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:29.879788 5882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:29.881060 5882 hybrid_clock.cc:648] HybridClock initialized: now 1790304629881001 us; error 50 us; skew 500 ppm
I20260925 02:50:29.881584 5882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:29 dist-test-slave-g1xh krb5kdc[5868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:29.892460 5882 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:50:29.894321 5882 webserver.cc:460] Webserver started at http://127.2.172.126:40027/ using document root <none> and password file <none>
I20260925 02:50:29.895172 5882 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:29.895284 5882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:29.895648 5882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:29.899363 5882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "7ed2be8effb24820959e0dd8d7a40443"
format_stamp: "Formatted at 2026-09-25 02:50:29 on dist-test-slave-g1xh"
server_key: "c65635ab536e050756cb2ade37d85121"
server_key_iv: "20eb6eaba1dcd3707445edc410d8fad2"
server_key_version: "encryptionkey@0"
I20260925 02:50:29.900252 5882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "7ed2be8effb24820959e0dd8d7a40443"
format_stamp: "Formatted at 2026-09-25 02:50:29 on dist-test-slave-g1xh"
server_key: "c65635ab536e050756cb2ade37d85121"
server_key_iv: "20eb6eaba1dcd3707445edc410d8fad2"
server_key_version: "encryptionkey@0"
I20260925 02:50:29.906615 5882 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:50:29.911059 5897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:29.913089 5882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:50:29.913314 5882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "7ed2be8effb24820959e0dd8d7a40443"
format_stamp: "Formatted at 2026-09-25 02:50:29 on dist-test-slave-g1xh"
server_key: "c65635ab536e050756cb2ade37d85121"
server_key_iv: "20eb6eaba1dcd3707445edc410d8fad2"
server_key_version: "encryptionkey@0"
I20260925 02:50:29.913537 5882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:29.929884 5882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:29.935199 5882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:29.935624 5882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:29.957931 5882 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:35695
I20260925 02:50:29.957978 5949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:35695 every 8 connection(s)
I20260925 02:50:29.959820 5882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:29.963754 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 5882
I20260925 02:50:29.964054 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:29.964761 2737 external_mini_cluster.cc:1428] Setting key ec7c1f8179442f2d7ce100f41df27b0b
I20260925 02:50:29.965044 5950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:29.982653 5950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443: Bootstrap starting.
Sep 25 02:50:29 dist-test-slave-g1xh krb5kdc[5868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:29.987988 5950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:29.989741 5950 log.cc:824] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:29.994505 5950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443: No bootstrap required, opened a new log
I20260925 02:50:29.995087 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.967338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.2.172.126:35695)
0925 02:50:29.968415 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:29.968436 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:29.968507 (+ 71us) server_negotiation.cc:407] Connection header received
0925 02:50:29.970942 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:29.970984 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:29.971760 (+ 776us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:29.972433 (+ 673us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:29.974028 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.976250 (+ 2222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.977713 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.978945 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.979175 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:29.984211 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:29.984227 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:29.984252 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:29.984313 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:29.989075 (+ 4762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:29.990148 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:29.990159 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:29.990169 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:29.990444 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:29.991084 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:29.991088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:29.991091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:29.992031 (+ 940us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:29.992380 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:29.992797 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 02:50:29.993358 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":495,"threads_started":1}
I20260925 02:50:30.004300 5950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed2be8effb24820959e0dd8d7a40443" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 35695 } }
I20260925 02:50:30.004779 5950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:30.004873 5950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed2be8effb24820959e0dd8d7a40443, State: Initialized, Role: FOLLOWER
I20260925 02:50:30.005646 5950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed2be8effb24820959e0dd8d7a40443" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 35695 } }
I20260925 02:50:30.005926 5950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:30.006109 5950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:30.006322 5950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:30.008935 5950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed2be8effb24820959e0dd8d7a40443" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 35695 } }
I20260925 02:50:30.009591 5950 leader_election.cc:302] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ed2be8effb24820959e0dd8d7a40443; no voters:
I20260925 02:50:30.010447 5950 leader_election.cc:288] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:30.010708 5955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:30.011781 5955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [term 1 LEADER]: Becoming Leader. State: Replica: 7ed2be8effb24820959e0dd8d7a40443, State: Running, Role: LEADER
I20260925 02:50:30.012430 5955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed2be8effb24820959e0dd8d7a40443" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 35695 } }
I20260925 02:50:30.013439 5950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:30.016036 5956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ed2be8effb24820959e0dd8d7a40443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed2be8effb24820959e0dd8d7a40443" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 35695 } } }
I20260925 02:50:30.016731 5956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:30.016127 5957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ed2be8effb24820959e0dd8d7a40443. Latest consensus state: current_term: 1 leader_uuid: "7ed2be8effb24820959e0dd8d7a40443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed2be8effb24820959e0dd8d7a40443" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 35695 } } }
I20260925 02:50:30.017865 5957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:30.019645 5963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:30.026139 5963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:30.036788 5963 catalog_manager.cc:1347] Generated new cluster ID: 8066a897011448fd8c85934d76d9c1ac
I20260925 02:50:30.036953 5963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:30.044382 5963 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:30.046710 5963 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:30.067902 5963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ed2be8effb24820959e0dd8d7a40443: Generated new TSK 0
I20260925 02:50:30.069393 5963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:50:30.136785 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:35695
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:30.542312 5976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:30.542654 5976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:30.542819 5976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:30.550544 5976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:30.550670 5976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:30.550854 5976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:50:30.561453 5976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:35695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:30.563010 5976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:30.564553 5976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:30.579206 5984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:30.579429 5982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:30.579586 5981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:30.580680 5976 server_base.cc:1034] running on GCE node
I20260925 02:50:30.581359 5976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:30.582523 5976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:30.583781 5976 hybrid_clock.cc:648] HybridClock initialized: now 1790304630583724 us; error 81 us; skew 500 ppm
I20260925 02:50:30.584142 5976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:30 dist-test-slave-g1xh krb5kdc[5868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304630, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:30.587850 5976 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:50:30.589624 5976 webserver.cc:460] Webserver started at http://127.2.172.65:38573/ using document root <none> and password file <none>
I20260925 02:50:30.590456 5976 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:30.590615 5976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:30.590942 5976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:30.594218 5976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "3dc8217e5dc64ba3939aaaf82dcbeea2"
format_stamp: "Formatted at 2026-09-25 02:50:30 on dist-test-slave-g1xh"
server_key: "a1ba6a47717ba4b629d33fffb6dab5fd"
server_key_iv: "c85225320e190bc835b9faf425c7b49d"
server_key_version: "encryptionkey@0"
I20260925 02:50:30.595081 5976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "3dc8217e5dc64ba3939aaaf82dcbeea2"
format_stamp: "Formatted at 2026-09-25 02:50:30 on dist-test-slave-g1xh"
server_key: "a1ba6a47717ba4b629d33fffb6dab5fd"
server_key_iv: "c85225320e190bc835b9faf425c7b49d"
server_key_version: "encryptionkey@0"
I20260925 02:50:30.600989 5976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 02:50:30.604921 5991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:30.606477 5976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:50:30.606681 5976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "3dc8217e5dc64ba3939aaaf82dcbeea2"
format_stamp: "Formatted at 2026-09-25 02:50:30 on dist-test-slave-g1xh"
server_key: "a1ba6a47717ba4b629d33fffb6dab5fd"
server_key_iv: "c85225320e190bc835b9faf425c7b49d"
server_key_version: "encryptionkey@0"
I20260925 02:50:30.606895 5976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:30.627199 5976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:30.631632 5976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:30.631955 5976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:30.632998 5976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:30.634483 5976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:30.634589 5976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:30.634749 5976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:30.634822 5976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:30.671171 5976 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:35103
I20260925 02:50:30.671228 6106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:35103 every 8 connection(s)
I20260925 02:50:30.672627 5976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:30.676700 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 5976
I20260925 02:50:30.676949 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:30.677856 2737 external_mini_cluster.cc:1428] Setting key 8b90406d5b518e9c03f915d59cf09fd7
Sep 25 02:50:30 dist-test-slave-g1xh krb5kdc[5868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304630, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:30.697688 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.675631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:52341 (local address 127.2.172.126:35695)
0925 02:50:30.676196 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:30.676205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:30.677169 (+ 964us) server_negotiation.cc:407] Connection header received
0925 02:50:30.679627 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:30.679631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:30.679703 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:30.679799 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:30.681963 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.682781 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.685439 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.685834 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.685941 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.689797 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:30.689807 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:30.689815 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:30.689866 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:30.693851 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.694695 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.694703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.694709 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.694865 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.695333 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.695341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.695348 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.695783 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:30.695948 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:30.696811 (+ 863us) server_negotiation.cc:299] Negotiation successful
0925 02:50:30.697144 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":280,"threads_started":1}
I20260925 02:50:30.698287 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.676069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:35695 (local address 127.2.172.65:52341)
0925 02:50:30.676961 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:30.677030 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:30.679256 (+ 2226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:30.680098 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:30.680108 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:30.680873 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:30.681745 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:30.681786 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.682961 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.682969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:30.685201 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:30.685220 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.686088 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.686098 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:30.686432 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.687231 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:30.687259 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:30.689502 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:30.694032 (+ 4530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:30.694041 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:30.694057 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:30.694536 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:30.694984 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:30.694988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:30.694990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:30.695185 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:30.695924 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:30.695931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:30.696355 (+ 424us) client_negotiation.cc:769] Sending connection context
0925 02:50:30.696634 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 02:50:30.697083 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":308,"threads_started":1}
I20260925 02:50:30.700147 6107 heartbeater.cc:347] Connected to a master server at 127.2.172.126:35695
I20260925 02:50:30.700527 6107 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
I20260925 02:50:30.701365 6107 heartbeater.cc:511] Master 127.2.172.126:35695 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
I20260925 02:50:30.703848 5914 ts_manager.cc:194] Registered new tserver with Master: 3dc8217e5dc64ba3939aaaf82dcbeea2 (127.2.172.65:35103)
I20260925 02:50:30.705422 5914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:52341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:50:30.730697 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:35695
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:31.094295 6113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:31.094630 6113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:31.094727 6113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:31.102154 6113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:31.102283 6113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:31.102397 6113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:50:31.113370 6113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:35695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:31.114921 6113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:31.116318 6113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:31.130475 6118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:31.130398 6121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:31.130439 6119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:31.131261 6113 server_base.cc:1034] running on GCE node
I20260925 02:50:31.131986 6113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:31.132946 6113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:31.134159 6113 hybrid_clock.cc:648] HybridClock initialized: now 1790304631134089 us; error 58 us; skew 500 ppm
I20260925 02:50:31.134456 6113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:31 dist-test-slave-g1xh krb5kdc[5868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:31.137753 6113 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:50:31.139348 6113 webserver.cc:460] Webserver started at http://127.2.172.66:41373/ using document root <none> and password file <none>
I20260925 02:50:31.140112 6113 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:31.140228 6113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:31.140514 6113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:31.143612 6113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "57c99b0dde0d478fae33b9eaee840953"
format_stamp: "Formatted at 2026-09-25 02:50:31 on dist-test-slave-g1xh"
server_key: "0e10f4d6a8f0a4adfa73ca3d58e3fb72"
server_key_iv: "66f0be4be825c458d5405c6bfaf85883"
server_key_version: "encryptionkey@0"
I20260925 02:50:31.144426 6113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "57c99b0dde0d478fae33b9eaee840953"
format_stamp: "Formatted at 2026-09-25 02:50:31 on dist-test-slave-g1xh"
server_key: "0e10f4d6a8f0a4adfa73ca3d58e3fb72"
server_key_iv: "66f0be4be825c458d5405c6bfaf85883"
server_key_version: "encryptionkey@0"
I20260925 02:50:31.149712 6113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:50:31.153195 6129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:31.154626 6113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:50:31.154824 6113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "57c99b0dde0d478fae33b9eaee840953"
format_stamp: "Formatted at 2026-09-25 02:50:31 on dist-test-slave-g1xh"
server_key: "0e10f4d6a8f0a4adfa73ca3d58e3fb72"
server_key_iv: "66f0be4be825c458d5405c6bfaf85883"
server_key_version: "encryptionkey@0"
I20260925 02:50:31.155028 6113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:31.176537 6113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:31.180907 6113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:31.181216 6113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:31.182296 6113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:31.183708 6113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:31.183811 6113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:31.183971 6113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:31.184046 6113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:31.220055 6113 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:38967
I20260925 02:50:31.220093 6244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:38967 every 8 connection(s)
I20260925 02:50:31.221329 6113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:31.222714 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 6113
I20260925 02:50:31.222936 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:31.223450 2737 external_mini_cluster.cc:1428] Setting key 243adefc82da8e87d059e01772c9d158
Sep 25 02:50:31 dist-test-slave-g1xh krb5kdc[5868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
I20260925 02:50:31.242442 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:31.224169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:57085 (local address 127.2.172.126:35695)
0925 02:50:31.224686 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:31.224691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:31.225777 (+ 1086us) server_negotiation.cc:407] Connection header received
0925 02:50:31.227729 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:31.227735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:31.227830 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:31.227952 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:31.229494 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.230197 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.231836 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.232264 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.232414 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:31.236167 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:31.236172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:31.236175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:31.236209 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:31.239114 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:31.240008 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:31.240012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:31.240015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:31.240155 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:31.240638 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:31.240642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:31.240644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:31.240915 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:31.241084 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:31.241806 (+ 722us) server_negotiation.cc:299] Negotiation successful
0925 02:50:31.242042 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":297,"threads_started":1}
I20260925 02:50:31.243532 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:31.224829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:35695 (local address 127.2.172.66:57085)
0925 02:50:31.225614 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:31.225654 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:31.227368 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:31.228207 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:31.228215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:31.228729 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:31.229317 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:31.229334 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.230374 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.230379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:31.231674 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:31.231687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.232509 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.232515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:31.232742 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:31.233434 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:31.233462 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:31.235902 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:31.239283 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:31.239293 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:31.239311 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:31.239868 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:31.240296 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:31.240301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:31.240305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:31.240504 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:31.241071 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:31.241079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:31.241435 (+ 356us) client_negotiation.cc:769] Sending connection context
0925 02:50:31.241701 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 02:50:31.242272 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":314,"threads_started":1}
I20260925 02:50:31.245041 6245 heartbeater.cc:347] Connected to a master server at 127.2.172.126:35695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
I20260925 02:50:31.245383 6245 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:31.246119 6245 heartbeater.cc:511] Master 127.2.172.126:35695 requested a full tablet report, sending...
I20260925 02:50:31.247617 5914 ts_manager.cc:194] Registered new tserver with Master: 57c99b0dde0d478fae33b9eaee840953 (127.2.172.66:38967)
I20260925 02:50:31.248461 5914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:57085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:50:31.271842 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:35695
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:31.637916 6251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:31.638211 6251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:31.638365 6251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:31.645979 6251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:31.646100 6251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:31.646273 6251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:50:31.657210 6251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:35695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:31.659061 6251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:31.660640 6251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:31.676132 6259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:31.676501 6257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:31.676249 6256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:31.677665 6251 server_base.cc:1034] running on GCE node
I20260925 02:50:31.678406 6251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:31.679448 6251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:31.680702 6251 hybrid_clock.cc:648] HybridClock initialized: now 1790304631680650 us; error 42 us; skew 500 ppm
I20260925 02:50:31.681094 6251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:31 dist-test-slave-g1xh krb5kdc[5868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:31.684556 6251 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:50:31.686142 6251 webserver.cc:460] Webserver started at http://127.2.172.67:38905/ using document root <none> and password file <none>
I20260925 02:50:31.686818 6251 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:31.686928 6251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:31.687179 6251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:31.690308 6251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "7ad444baa25249f48852eb394558ebe8"
format_stamp: "Formatted at 2026-09-25 02:50:31 on dist-test-slave-g1xh"
server_key: "3d887e8ccdf7c3e3bb37e994eba0d400"
server_key_iv: "2dfd3ab4dd3f761846cfc80cb4d08075"
server_key_version: "encryptionkey@0"
I20260925 02:50:31.691220 6251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "7ad444baa25249f48852eb394558ebe8"
format_stamp: "Formatted at 2026-09-25 02:50:31 on dist-test-slave-g1xh"
server_key: "3d887e8ccdf7c3e3bb37e994eba0d400"
server_key_iv: "2dfd3ab4dd3f761846cfc80cb4d08075"
server_key_version: "encryptionkey@0"
I20260925 02:50:31.696921 6251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:50:31.700754 6266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:31.702283 6251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:50:31.702463 6251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "7ad444baa25249f48852eb394558ebe8"
format_stamp: "Formatted at 2026-09-25 02:50:31 on dist-test-slave-g1xh"
server_key: "3d887e8ccdf7c3e3bb37e994eba0d400"
server_key_iv: "2dfd3ab4dd3f761846cfc80cb4d08075"
server_key_version: "encryptionkey@0"
I20260925 02:50:31.702674 6251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:31.708604 6107 heartbeater.cc:503] Master 127.2.172.126:35695 was elected leader, sending a full tablet report...
I20260925 02:50:31.714412 6251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:31.719038 6251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:31.719362 6251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:31.720527 6251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:31.721952 6251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:31.722064 6251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:31.722218 6251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:31.722290 6251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:31.756942 6251 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:45237
I20260925 02:50:31.756986 6380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:45237 every 8 connection(s)
I20260925 02:50:31.758307 6251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:31.763368 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 6251
I20260925 02:50:31.763562 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:31.764063 2737 external_mini_cluster.cc:1428] Setting key 17a254a6e7dde9c9911dc3bec18afe2a
Sep 25 02:50:31 dist-test-slave-g1xh krb5kdc[5868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:31.779609 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:31.761028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:57861 (local address 127.2.172.126:35695)
0925 02:50:31.761496 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:31.761501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:31.762440 (+ 939us) server_negotiation.cc:407] Connection header received
0925 02:50:31.764536 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:31.764542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:31.764637 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:31.764766 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:31.766657 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.767597 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.769126 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.769510 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.769603 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:31.772988 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:31.772993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:31.772996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:31.773030 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:31.776051 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:31.776828 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:31.776832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:31.776835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:31.776958 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:31.777536 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:31.777541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:31.777544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:31.777882 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:31.778055 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:31.779043 (+ 988us) server_negotiation.cc:299] Negotiation successful
0925 02:50:31.779274 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":223,"threads_started":1}
I20260925 02:50:31.780692 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:31.761604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:35695 (local address 127.2.172.67:57861)
0925 02:50:31.762282 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:31.762320 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:31.764257 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:31.765032 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:31.765043 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:31.765712 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:31.766485 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:31.766510 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.767736 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.767740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:31.768971 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:31.768983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.769689 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.769693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:31.769897 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:31.770571 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:31.770592 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:31.772740 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:31.776207 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:31.776214 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:31.776230 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:31.776716 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:31.777112 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:31.777116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:31.777119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:31.777382 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:31.778063 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:31.778070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:31.778547 (+ 477us) client_negotiation.cc:769] Sending connection context
0925 02:50:31.778887 (+ 340us) client_negotiation.cc:240] Negotiation successful
0925 02:50:31.779413 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":275,"threads_started":1}
I20260925 02:50:31.781972 6381 heartbeater.cc:347] Connected to a master server at 127.2.172.126:35695
I20260925 02:50:31.782258 6381 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:31.782904 6381 heartbeater.cc:511] Master 127.2.172.126:35695 requested a full tablet report, sending...
I20260925 02:50:31.784273 5914 ts_manager.cc:194] Registered new tserver with Master: 7ad444baa25249f48852eb394558ebe8 (127.2.172.67:45237)
I20260925 02:50:31.785022 5914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:57861
I20260925 02:50:31.792248 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:50:31 dist-test-slave-g1xh krb5kdc[5868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304631, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 02:50:31 dist-test-slave-g1xh krb5kdc[5868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304631, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:31.844293 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:31.829571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.2.172.126:35695)
0925 02:50:31.829934 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:31.829943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:31.830294 (+ 351us) server_negotiation.cc:407] Connection header received
0925 02:50:31.830535 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:31.830540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:31.830607 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:31.830717 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:31.832147 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.832887 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.834139 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.834587 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.834729 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:31.838736 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:31.838742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:31.838747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:31.838791 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:31.841471 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:31.842184 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:31.842191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:31.842195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:31.842341 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:31.842717 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:31.842722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:31.842725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:31.843056 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:31.843317 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:31.843684 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 02:50:31.843910 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
W20260925 02:50:31.844974 5914 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46860
I20260925 02:50:31.847076 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 5976
Sep 25 02:50:31 dist-test-slave-g1xh krb5kdc[5868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304631, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:31.863354 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:31.848125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.2.172.65:35103)
0925 02:50:31.848852 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:31.848859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:31.848883 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:50:31.848939 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:31.848945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:31.849152 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:31.849278 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:31.850482 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.851479 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.852764 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.853329 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.853429 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:31.857528 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:31.857542 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:31.857557 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:31.857592 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:31.860675 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:31.861372 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:31.861380 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:31.861382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:31.861533 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:31.861909 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:31.861913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:31.861915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:31.862316 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:31.862560 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:31.862858 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 02:50:31.863062 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":424,"threads_started":1}
W20260925 02:50:31.864699 6102 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53640
W20260925 02:50:31.865934 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 5976: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:50:31.866153 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 5976
I20260925 02:50:31.884336 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 6113
Sep 25 02:50:31 dist-test-slave-g1xh krb5kdc[5868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304631, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:50:31.901409 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:31.885260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.2.172.66:38967)
0925 02:50:31.885775 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:31.885782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:31.885807 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:50:31.885875 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:31.885881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:31.886100 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:31.886235 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:31.887654 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.888655 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.889922 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.890461 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.890558 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:31.895077 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:31.895090 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:31.895106 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:31.895141 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:31.898622 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:31.899303 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:31.899310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:31.899312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:31.899434 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:31.899873 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:31.899876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:31.899878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:31.900226 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:31.900468 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:31.900778 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 02:50:31.901005 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":254,"threads_started":1}
W20260925 02:50:31.902644 6240 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52090
W20260925 02:50:31.903733 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 6113: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:50:31.903944 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 6113
I20260925 02:50:31.921718 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 6251
Sep 25 02:50:31 dist-test-slave-g1xh krb5kdc[5868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304631, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:50:31.938876 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:31.922707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.2.172.67:45237)
0925 02:50:31.923368 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:31.923375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:31.923406 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:50:31.923466 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:31.923472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:31.923775 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:31.923956 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:31.925322 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.926356 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.927662 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:31.928221 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:31.928401 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:31.933020 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:31.933062 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:31.933078 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:31.933116 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:31.936246 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:31.936921 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:31.936928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:31.936931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:31.937047 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:31.937397 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:31.937399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:31.937401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:31.937727 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:31.937961 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:31.938220 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 02:50:31.938502 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":345,"threads_started":1}
W20260925 02:50:31.940212 6376 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33876
W20260925 02:50:31.941368 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 6251: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:50:31.941576 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 6251
I20260925 02:50:31.959406 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 5882
I20260925 02:50:31.960719 5944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:32.066452 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 5882
2026-09-25T02:50:32Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3965 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:32 dist-test-slave-g1xh krb5kdc[6399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:32 dist-test-slave-g1xh krb5kdc[6399](info): set up 2 sockets
Sep 25 02:50:32 dist-test-slave-g1xh krb5kdc[6399](info): commencing operation
krb5kdc: starting...
W20260925 02:50:33.175279 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:50:33 dist-test-slave-g1xh krb5kdc[6399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:50:33.396638 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:39641
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:39641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:33.778461 6412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:33.778923 6412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:33.779111 6412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:33.787519 6412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:33.787781 6412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:33.787909 6412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:33.788017 6412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:33.788113 6412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:33.799909 6412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:39641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:39641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:33.801973 6412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:33.803761 6412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:33.819293 6420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:33.819262 6418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:33.819228 6417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:33.820474 6412 server_base.cc:1034] running on GCE node
I20260925 02:50:33.821497 6412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:33.823346 6412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:33.824681 6412 hybrid_clock.cc:648] HybridClock initialized: now 1790304633824630 us; error 68 us; skew 500 ppm
I20260925 02:50:33.825197 6412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:33 dist-test-slave-g1xh krb5kdc[6399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304633, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:33.829187 6412 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:50:33.831137 6412 webserver.cc:460] Webserver started at http://127.2.172.126:44133/ using document root <none> and password file <none>
I20260925 02:50:33.832162 6412 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:33.832288 6412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:33.832659 6412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:33.836802 6412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "b19c459c6c0341978fcb47cbd6794f96"
format_stamp: "Formatted at 2026-09-25 02:50:33 on dist-test-slave-g1xh"
server_key: "c93964206468eaf9908e52c1f51d59fe"
server_key_iv: "0a2d2c8bf542a362ed46696c2533149c"
server_key_version: "encryptionkey@0"
I20260925 02:50:33.837723 6412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "b19c459c6c0341978fcb47cbd6794f96"
format_stamp: "Formatted at 2026-09-25 02:50:33 on dist-test-slave-g1xh"
server_key: "c93964206468eaf9908e52c1f51d59fe"
server_key_iv: "0a2d2c8bf542a362ed46696c2533149c"
server_key_version: "encryptionkey@0"
I20260925 02:50:33.844419 6412 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 02:50:33.848939 6427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:33.850841 6412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:50:33.851034 6412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "b19c459c6c0341978fcb47cbd6794f96"
format_stamp: "Formatted at 2026-09-25 02:50:33 on dist-test-slave-g1xh"
server_key: "c93964206468eaf9908e52c1f51d59fe"
server_key_iv: "0a2d2c8bf542a362ed46696c2533149c"
server_key_version: "encryptionkey@0"
I20260925 02:50:33.851272 6412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:33.899072 6412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:33.905161 6412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:33.905484 6412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:33.929875 6412 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:39641
I20260925 02:50:33.929894 6479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:39641 every 8 connection(s)
I20260925 02:50:33.931936 6412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:33.937264 6480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:33.942260 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 6412
I20260925 02:50:33.942518 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:33.943463 2737 external_mini_cluster.cc:1428] Setting key e3134e0a4e42c0d3baa478ebdf3773d4
I20260925 02:50:33.952052 6480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96: Bootstrap starting.
I20260925 02:50:33.956817 6480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:33.958283 6480 log.cc:824] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:50:33 dist-test-slave-g1xh krb5kdc[6399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:33.962445 6480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96: No bootstrap required, opened a new log
I20260925 02:50:33.971592 6480 raft_consensus.cc:348] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19c459c6c0341978fcb47cbd6794f96" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39641 } }
I20260925 02:50:33.972010 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:33.946288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.2.172.126:39641)
0925 02:50:33.947256 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:33.947271 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:33.947326 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:50:33.950052 (+ 2726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:33.950085 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:33.950615 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:33.951027 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:33.953250 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:33.954770 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:33.956723 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:33.957740 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:33.957899 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:33.963222 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:33.963233 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:33.963251 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:33.963299 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:33.966904 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:33.967806 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:33.967819 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:33.967828 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:33.968028 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:33.968526 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:33.968536 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:33.968538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:33.969237 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:33.969638 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:33.970114 (+ 476us) server_negotiation.cc:299] Negotiation successful
0925 02:50:33.970667 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":459,"threads_started":1}
I20260925 02:50:33.972131 6480 raft_consensus.cc:374] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:33.972254 6480 raft_consensus.cc:729] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19c459c6c0341978fcb47cbd6794f96, State: Initialized, Role: FOLLOWER
I20260925 02:50:33.973084 6480 consensus_queue.cc:260] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19c459c6c0341978fcb47cbd6794f96" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39641 } }
I20260925 02:50:33.973389 6480 raft_consensus.cc:388] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:33.973541 6480 raft_consensus.cc:482] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:33.973754 6480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:33.976702 6480 raft_consensus.cc:504] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19c459c6c0341978fcb47cbd6794f96" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39641 } }
I20260925 02:50:33.977278 6480 leader_election.cc:302] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b19c459c6c0341978fcb47cbd6794f96; no voters:
I20260925 02:50:33.978011 6480 leader_election.cc:288] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:33.978247 6486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:33.979125 6486 raft_consensus.cc:686] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [term 1 LEADER]: Becoming Leader. State: Replica: b19c459c6c0341978fcb47cbd6794f96, State: Running, Role: LEADER
I20260925 02:50:33.979750 6486 consensus_queue.cc:237] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19c459c6c0341978fcb47cbd6794f96" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39641 } }
I20260925 02:50:33.980799 6480 sys_catalog.cc:564] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:33.983605 6488 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b19c459c6c0341978fcb47cbd6794f96. Latest consensus state: current_term: 1 leader_uuid: "b19c459c6c0341978fcb47cbd6794f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19c459c6c0341978fcb47cbd6794f96" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39641 } } }
I20260925 02:50:33.983927 6487 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b19c459c6c0341978fcb47cbd6794f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19c459c6c0341978fcb47cbd6794f96" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39641 } } }
I20260925 02:50:33.984143 6488 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:33.984249 6487 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:33.986284 6495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:33.992897 6495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:34.004492 6495 catalog_manager.cc:1347] Generated new cluster ID: 5194fb561b6a4103975d97b920487851
I20260925 02:50:34.004662 6495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:34.030951 6495 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:34.033004 6495 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:34.041432 6495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b19c459c6c0341978fcb47cbd6794f96: Generated new TSK 0
I20260925 02:50:34.042528 6495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:50:34.112452 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:39641
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:34.522437 6507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:34.522861 6507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:34.523044 6507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:34.531404 6507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:34.531615 6507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:34.531888 6507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:50:34.543885 6507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:39641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:34.546252 6507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:34.548306 6507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:34.564445 6513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:34.564559 6512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:34.564556 6515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:34.565481 6507 server_base.cc:1034] running on GCE node
I20260925 02:50:34.566119 6507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:34.567137 6507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:34.568361 6507 hybrid_clock.cc:648] HybridClock initialized: now 1790304634568280 us; error 62 us; skew 500 ppm
I20260925 02:50:34.568769 6507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:34 dist-test-slave-g1xh krb5kdc[6399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304634, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:34.572227 6507 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:50:34.573915 6507 webserver.cc:460] Webserver started at http://127.2.172.65:33087/ using document root <none> and password file <none>
I20260925 02:50:34.574714 6507 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:34.574868 6507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:34.575199 6507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:34.578435 6507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "c1e245788a4842f894b68873dfd0f94a"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-g1xh"
server_key: "de73e460e5d9693e38703b1d092e9460"
server_key_iv: "819bc1825666cebfbef636cfde4ac4bf"
server_key_version: "encryptionkey@0"
I20260925 02:50:34.579394 6507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "c1e245788a4842f894b68873dfd0f94a"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-g1xh"
server_key: "de73e460e5d9693e38703b1d092e9460"
server_key_iv: "819bc1825666cebfbef636cfde4ac4bf"
server_key_version: "encryptionkey@0"
I20260925 02:50:34.586197 6507 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:50:34.590224 6522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:34.591809 6507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:34.592001 6507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "c1e245788a4842f894b68873dfd0f94a"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-g1xh"
server_key: "de73e460e5d9693e38703b1d092e9460"
server_key_iv: "819bc1825666cebfbef636cfde4ac4bf"
server_key_version: "encryptionkey@0"
I20260925 02:50:34.592228 6507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:34.622205 6507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:34.626977 6507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:34.627316 6507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:34.628384 6507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:34.629895 6507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:34.630004 6507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:34.630155 6507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:34.630228 6507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:34.665679 6507 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:39103
I20260925 02:50:34.665728 6635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:39103 every 8 connection(s)
I20260925 02:50:34.667055 6507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:34.670913 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 6507
I20260925 02:50:34.671176 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:34.672211 2737 external_mini_cluster.cc:1428] Setting key f459ce4acff34314125a11372304be4a
Sep 25 02:50:34 dist-test-slave-g1xh krb5kdc[6399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304634, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:34.689769 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:34.669860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:38637 (local address 127.2.172.126:39641)
0925 02:50:34.670387 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:34.670392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:34.672280 (+ 1888us) server_negotiation.cc:407] Connection header received
0925 02:50:34.674544 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:34.674550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:34.674643 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:34.674762 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:34.676440 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:34.677197 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:34.678804 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:34.679223 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:34.679364 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:34.683469 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:34.683475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:34.683479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:34.683514 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:34.686460 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:34.687239 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:34.687244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:34.687246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:34.687374 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:34.687911 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:34.687923 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:34.687926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:34.688194 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:34.688360 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:34.689107 (+ 747us) server_negotiation.cc:299] Negotiation successful
0925 02:50:34.689338 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":276,"threads_started":1}
I20260925 02:50:34.690819 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:34.670355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39641 (local address 127.2.172.65:38637)
0925 02:50:34.672101 (+ 1746us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:34.672148 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:34.674249 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:34.675010 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:34.675018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:34.675663 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:34.676265 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:34.676282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:34.677365 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:34.677369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:34.678652 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:34.678664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:34.679487 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:34.679496 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:34.679770 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:34.680499 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:34.680524 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:34.683221 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:34.686610 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:34.686619 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:34.686639 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:34.687096 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:34.687501 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:34.687506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:34.687510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:34.687784 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:34.688330 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:34.688339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:34.688717 (+ 378us) client_negotiation.cc:769] Sending connection context
0925 02:50:34.688995 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 02:50:34.689590 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1494,"thread_start_us":1286,"threads_started":1}
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
I20260925 02:50:34.692512 6636 heartbeater.cc:347] Connected to a master server at 127.2.172.126:39641
I20260925 02:50:34.692867 6636 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:34.693640 6636 heartbeater.cc:511] Master 127.2.172.126:39641 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
I20260925 02:50:34.696148 6444 ts_manager.cc:194] Registered new tserver with Master: c1e245788a4842f894b68873dfd0f94a (127.2.172.65:39103)
I20260925 02:50:34.697714 6444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:38637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:50:34.719177 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:39641
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:35.074693 6643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:35.074977 6643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:35.075145 6643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:35.082537 6643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:35.082665 6643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:35.082913 6643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:50:35.093469 6643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:39641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:35.094813 6643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:35.096304 6643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:35.110265 6648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:35.110314 6649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:35.110379 6651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:35.111094 6643 server_base.cc:1034] running on GCE node
I20260925 02:50:35.111825 6643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:35.112833 6643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:35.114048 6643 hybrid_clock.cc:648] HybridClock initialized: now 1790304635113985 us; error 54 us; skew 500 ppm
I20260925 02:50:35.114415 6643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:35 dist-test-slave-g1xh krb5kdc[6399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:35.124078 6643 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:50:35.125684 6643 webserver.cc:460] Webserver started at http://127.2.172.66:41753/ using document root <none> and password file <none>
I20260925 02:50:35.126381 6643 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:35.126488 6643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:35.126735 6643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:35.129752 6643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "7c0f427c6c5a4e4898590a2d52da2d56"
format_stamp: "Formatted at 2026-09-25 02:50:35 on dist-test-slave-g1xh"
server_key: "5b763c6a8aaf4588ea3891968d2059aa"
server_key_iv: "2b97776e289490e8acde1accc8d83ec9"
server_key_version: "encryptionkey@0"
I20260925 02:50:35.130493 6643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "7c0f427c6c5a4e4898590a2d52da2d56"
format_stamp: "Formatted at 2026-09-25 02:50:35 on dist-test-slave-g1xh"
server_key: "5b763c6a8aaf4588ea3891968d2059aa"
server_key_iv: "2b97776e289490e8acde1accc8d83ec9"
server_key_version: "encryptionkey@0"
I20260925 02:50:35.135776 6643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 02:50:35.139340 6658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.140856 6643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:35.141041 6643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "7c0f427c6c5a4e4898590a2d52da2d56"
format_stamp: "Formatted at 2026-09-25 02:50:35 on dist-test-slave-g1xh"
server_key: "5b763c6a8aaf4588ea3891968d2059aa"
server_key_iv: "2b97776e289490e8acde1accc8d83ec9"
server_key_version: "encryptionkey@0"
I20260925 02:50:35.141245 6643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:35.164578 6643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:35.169005 6643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:35.169306 6643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:35.170282 6643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:35.171675 6643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:35.171753 6643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.171904 6643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:35.171965 6643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.206245 6643 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:38539
I20260925 02:50:35.206302 6772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:38539 every 8 connection(s)
I20260925 02:50:35.207603 6643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:35.210217 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 6643
I20260925 02:50:35.210445 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:35.211020 2737 external_mini_cluster.cc:1428] Setting key 715c1640a0856fa2c012bbbca70a7380
Sep 25 02:50:35 dist-test-slave-g1xh krb5kdc[6399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:35.230175 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.210248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:54595 (local address 127.2.172.126:39641)
0925 02:50:35.210828 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.210834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.211898 (+ 1064us) server_negotiation.cc:407] Connection header received
0925 02:50:35.213761 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.213777 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.213867 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.213988 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:35.215741 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.216942 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.218535 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.219019 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.219121 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.223724 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:35.223730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:35.223734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:35.223785 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:35.226514 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.227621 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.227625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.227627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.227754 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.228343 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.228348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.228350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.228646 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:35.228765 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.229724 (+ 959us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.229920 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":234,"threads_started":1}
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
I20260925 02:50:35.231745 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.210864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39641 (local address 127.2.172.66:54595)
0925 02:50:35.211735 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:35.211787 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:35.213498 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:35.214179 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:35.214187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:35.214933 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:35.215506 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.215528 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.217104 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.217108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.218385 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.218397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.219282 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.219290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.219639 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.220387 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:35.220423 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:35.223391 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:35.226686 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:35.226699 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:35.226721 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:35.227428 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:35.227894 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:35.227901 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:35.227906 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:35.228183 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:35.228788 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:35.228798 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:35.229296 (+ 498us) client_negotiation.cc:769] Sending connection context
0925 02:50:35.229627 (+ 331us) client_negotiation.cc:240] Negotiation successful
0925 02:50:35.230222 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":345,"threads_started":1}
I20260925 02:50:35.233245 6773 heartbeater.cc:347] Connected to a master server at 127.2.172.126:39641
I20260925 02:50:35.233603 6773 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:35.234386 6773 heartbeater.cc:511] Master 127.2.172.126:39641 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
I20260925 02:50:35.235917 6444 ts_manager.cc:194] Registered new tserver with Master: 7c0f427c6c5a4e4898590a2d52da2d56 (127.2.172.66:38539)
I20260925 02:50:35.236815 6444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:54595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:50:35.259636 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:39641
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:35.599962 6779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:35.600279 6779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:35.600472 6779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:35.608531 6779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:35.608649 6779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:35.608955 6779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:50:35.619784 6779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:39641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:35.621145 6779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:35.622433 6779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:35.636715 6785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:35.636682 6787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:35.636662 6784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:35.637796 6779 server_base.cc:1034] running on GCE node
I20260925 02:50:35.638585 6779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:35.639664 6779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:35.640909 6779 hybrid_clock.cc:648] HybridClock initialized: now 1790304635640858 us; error 45 us; skew 500 ppm
I20260925 02:50:35.641264 6779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:35 dist-test-slave-g1xh krb5kdc[6399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:35.644861 6779 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:50:35.646284 6779 webserver.cc:460] Webserver started at http://127.2.172.67:40275/ using document root <none> and password file <none>
I20260925 02:50:35.646962 6779 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:35.647069 6779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:35.647317 6779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:35.650547 6779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "7efe9f859c7f454fac7513e46cfded33"
format_stamp: "Formatted at 2026-09-25 02:50:35 on dist-test-slave-g1xh"
server_key: "2239cb3ad1f8e9b55374518b0d433524"
server_key_iv: "d814d3f9cc678f88119d806dc2509ac8"
server_key_version: "encryptionkey@0"
I20260925 02:50:35.651362 6779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "7efe9f859c7f454fac7513e46cfded33"
format_stamp: "Formatted at 2026-09-25 02:50:35 on dist-test-slave-g1xh"
server_key: "2239cb3ad1f8e9b55374518b0d433524"
server_key_iv: "d814d3f9cc678f88119d806dc2509ac8"
server_key_version: "encryptionkey@0"
I20260925 02:50:35.657127 6779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:50:35.661458 6795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.663151 6779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:50:35.663358 6779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "7efe9f859c7f454fac7513e46cfded33"
format_stamp: "Formatted at 2026-09-25 02:50:35 on dist-test-slave-g1xh"
server_key: "2239cb3ad1f8e9b55374518b0d433524"
server_key_iv: "d814d3f9cc678f88119d806dc2509ac8"
server_key_version: "encryptionkey@0"
I20260925 02:50:35.663615 6779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:35.685021 6779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:35.689877 6779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:35.690205 6779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:35.691229 6779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:35.692693 6779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:35.692771 6779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.692929 6779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:35.692981 6779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.701272 6636 heartbeater.cc:503] Master 127.2.172.126:39641 was elected leader, sending a full tablet report...
I20260925 02:50:35.726106 6779 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:41715
I20260925 02:50:35.726155 6911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:41715 every 8 connection(s)
I20260925 02:50:35.727403 6779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:35.730656 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 6779
I20260925 02:50:35.730840 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:35.731287 2737 external_mini_cluster.cc:1428] Setting key 0813e110fbd2c39f795e7ba127691f0e
Sep 25 02:50:35 dist-test-slave-g1xh krb5kdc[6399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:35.749452 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.730022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:49885 (local address 127.2.172.126:39641)
0925 02:50:35.730203 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.730207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.731449 (+ 1242us) server_negotiation.cc:407] Connection header received
0925 02:50:35.733613 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.733618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.733687 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.733767 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:35.735397 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.736248 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.737823 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.738363 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.738463 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.742091 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:35.742097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:35.742100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:35.742137 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:35.745244 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.746661 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.746667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.746669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.746876 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.747665 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.747670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.747673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.748001 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:35.748127 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.748934 (+ 807us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.749129 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:50:35.750731 6914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.730411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39641 (local address 127.2.172.67:49885)
0925 02:50:35.731258 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:35.731322 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:35.733263 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:35.734059 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:35.734073 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:35.734653 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:35.735219 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.735241 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.736457 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.736461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.737689 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.737701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.738618 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.738641 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.738869 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.739494 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:35.739518 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:35.741827 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:35.745639 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:35.745648 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:35.745666 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:35.746484 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:35.747044 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:35.747049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:35.747052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:35.747478 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:35.748132 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:35.748139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:35.748567 (+ 428us) client_negotiation.cc:769] Sending connection context
0925 02:50:35.748812 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 02:50:35.749233 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":384,"threads_started":1}
I20260925 02:50:35.752367 6912 heartbeater.cc:347] Connected to a master server at 127.2.172.126:39641
I20260925 02:50:35.752715 6912 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:35.753531 6912 heartbeater.cc:511] Master 127.2.172.126:39641 requested a full tablet report, sending...
I20260925 02:50:35.755172 6444 ts_manager.cc:194] Registered new tserver with Master: 7efe9f859c7f454fac7513e46cfded33 (127.2.172.67:41715)
I20260925 02:50:35.756174 6444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:49885
I20260925 02:50:35.760373 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:50:35 dist-test-slave-g1xh krb5kdc[6399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:50:35 dist-test-slave-g1xh krb5kdc[6399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:35.814911 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.798135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.2.172.65:39103)
0925 02:50:35.798988 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.798995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.799028 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:50:35.799121 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.799127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.799345 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.799489 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:35.801096 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.802121 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.803625 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.804226 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.804400 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.808858 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:35.808905 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:35.808921 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:35.808968 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:35.812163 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.812917 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.812924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.812927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.813071 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.813474 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.813478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.813480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.813837 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:35.814073 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.814394 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.814615 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":575,"threads_started":1}
Sep 25 02:50:35 dist-test-slave-g1xh krb5kdc[6399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:35.834668 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.820412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.2.172.126:39641)
0925 02:50:35.820650 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.820658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.820794 (+ 136us) server_negotiation.cc:407] Connection header received
0925 02:50:35.820987 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.820992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.821070 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.821155 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:35.822590 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.823309 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.824724 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.825151 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.825249 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.829399 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:35.829405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:35.829408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:35.829444 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:35.832052 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.832707 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.832714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.832718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.832853 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.833198 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.833204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.833207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.833527 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:35.833797 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.834104 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.834326 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260925 02:50:35.835304 6444 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59306
I20260925 02:50:35.836577 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 6507
I20260925 02:50:35.850914 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.837543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.2.172.65:39103)
0925 02:50:35.837716 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.837721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.838121 (+ 400us) server_negotiation.cc:407] Connection header received
0925 02:50:35.838268 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.838275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.838415 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.838549 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:35.839981 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.840703 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.842344 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.842755 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.842861 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.845649 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:35.845655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:35.845658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:35.845693 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:35.848492 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.849157 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.849161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.849163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.849324 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.849647 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.849650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.849652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.849939 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:35.850108 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.850489 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.850661 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:50:35.851650 6631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:35.936193 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 6507
I20260925 02:50:35.957644 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 6643
Sep 25 02:50:35 dist-test-slave-g1xh krb5kdc[6399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:50:35.974928 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.958676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.2.172.66:38539)
0925 02:50:35.959201 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.959208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.959234 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:50:35.959346 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.959356 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.959682 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.959844 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:35.961178 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.962274 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.963617 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.964219 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.964350 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.968834 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:35.968847 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:35.968862 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:35.968903 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:35.971985 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.972748 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.972755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.972757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.972913 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.973374 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.973378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.973380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.973729 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:35.973984 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.974296 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.974501 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":286,"threads_started":1}
I20260925 02:50:35.976408 6768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:36.061012 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 6643
I20260925 02:50:36.080128 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 6779
Sep 25 02:50:36 dist-test-slave-g1xh krb5kdc[6399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:50:36.098587 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.081495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.2.172.67:41715)
0925 02:50:36.082158 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:36.082165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:36.082194 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:50:36.082254 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:36.082260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:36.082592 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:36.082773 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:36.084175 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.085340 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.086729 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.087326 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.087489 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.092261 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:36.092276 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:36.092291 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:36.092345 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:36.095448 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:36.096337 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:36.096345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:36.096347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:36.096532 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:36.096926 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:36.096930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:36.096932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:36.097302 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:36.097586 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:36.097920 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 02:50:36.098141 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":381,"threads_started":1}
I20260925 02:50:36.099927 6907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:36.184122 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 6779
I20260925 02:50:36.201910 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 6412
I20260925 02:50:36.203410 6475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:36.309353 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 6412
2026-09-25T02:50:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4242 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:36 dist-test-slave-g1xh krb5kdc[6931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:36 dist-test-slave-g1xh krb5kdc[6931](info): set up 2 sockets
Sep 25 02:50:36 dist-test-slave-g1xh krb5kdc[6931](info): commencing operation
krb5kdc: starting...
W20260925 02:50:37.419934 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:50:37 dist-test-slave-g1xh krb5kdc[6931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:50:37.696554 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:34789
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:34789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:38.083508 6944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:38.083956 6944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:38.084064 6944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:38.092389 6944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:38.092546 6944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:38.092600 6944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:38.092667 6944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:38.092720 6944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:38.104025 6944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:34789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:34789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:38.105783 6944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:38.107419 6944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:38.121274 6949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:38.121362 6950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:38.121333 6952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:38.122476 6944 server_base.cc:1034] running on GCE node
I20260925 02:50:38.123492 6944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:38.125348 6944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:38.126644 6944 hybrid_clock.cc:648] HybridClock initialized: now 1790304638126599 us; error 68 us; skew 500 ppm
I20260925 02:50:38.127132 6944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:38 dist-test-slave-g1xh krb5kdc[6931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304638, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:38.130916 6944 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:50:38.132512 6944 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:38.132665 6944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:38.133039 6944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:38.136744 6944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "cdd9c45cbacc42cd971371d05d41ce5f"
format_stamp: "Formatted at 2026-09-25 02:50:38 on dist-test-slave-g1xh"
server_key: "9e73ad896b46b79c880d533f1ab9e7e6"
server_key_iv: "5121a53baa0f8a378749ec0a5b4b2548"
server_key_version: "encryptionkey@0"
I20260925 02:50:38.137676 6944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "cdd9c45cbacc42cd971371d05d41ce5f"
format_stamp: "Formatted at 2026-09-25 02:50:38 on dist-test-slave-g1xh"
server_key: "9e73ad896b46b79c880d533f1ab9e7e6"
server_key_iv: "5121a53baa0f8a378749ec0a5b4b2548"
server_key_version: "encryptionkey@0"
I20260925 02:50:38.144565 6944 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260925 02:50:38.149120 6959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:38.150949 6944 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260925 02:50:38.151269 6944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "cdd9c45cbacc42cd971371d05d41ce5f"
format_stamp: "Formatted at 2026-09-25 02:50:38 on dist-test-slave-g1xh"
server_key: "9e73ad896b46b79c880d533f1ab9e7e6"
server_key_iv: "5121a53baa0f8a378749ec0a5b4b2548"
server_key_version: "encryptionkey@0"
I20260925 02:50:38.151551 6944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:38.169588 6944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:38.174697 6944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:38.175057 6944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:38.197829 6944 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:34789
I20260925 02:50:38.197850 7011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:34789 every 8 connection(s)
I20260925 02:50:38.199067 6944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:38.204174 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:38.205053 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 6944
I20260925 02:50:38.205314 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:38.206075 2737 external_mini_cluster.cc:1428] Setting key b45987a3416c9db6a22779153093cdcc
I20260925 02:50:38.222276 7012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f: Bootstrap starting.
Sep 25 02:50:38 dist-test-slave-g1xh krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:38.227182 7012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:38.228946 7012 log.cc:824] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:38.233018 7012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f: No bootstrap required, opened a new log
I20260925 02:50:38.235236 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:38.208620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.2.172.126:34789)
0925 02:50:38.209662 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:38.209677 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:38.209735 (+ 58us) server_negotiation.cc:407] Connection header received
0925 02:50:38.211722 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:38.211758 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:38.212345 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:38.212831 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:38.215212 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.217056 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.219170 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.220198 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.220380 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:38.225778 (+ 5398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:38.225790 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:38.225812 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:38.225882 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:38.230092 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:38.230954 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:38.230964 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:38.230972 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:38.231183 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:38.231599 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:38.231603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:38.231606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:38.232424 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:38.232744 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:38.233158 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 02:50:38.233699 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":518,"threads_started":1}
I20260925 02:50:38.242153 7012 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd9c45cbacc42cd971371d05d41ce5f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34789 } }
I20260925 02:50:38.242602 7012 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:38.242674 7012 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd9c45cbacc42cd971371d05d41ce5f, State: Initialized, Role: FOLLOWER
I20260925 02:50:38.243294 7012 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd9c45cbacc42cd971371d05d41ce5f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34789 } }
I20260925 02:50:38.243593 7012 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:38.243765 7012 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:38.243927 7012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:38.246887 7012 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd9c45cbacc42cd971371d05d41ce5f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34789 } }
I20260925 02:50:38.247622 7012 leader_election.cc:302] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdd9c45cbacc42cd971371d05d41ce5f; no voters:
I20260925 02:50:38.248301 7012 leader_election.cc:288] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:38.248478 7017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:38.249315 7017 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [term 1 LEADER]: Becoming Leader. State: Replica: cdd9c45cbacc42cd971371d05d41ce5f, State: Running, Role: LEADER
I20260925 02:50:38.249961 7017 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd9c45cbacc42cd971371d05d41ce5f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34789 } }
I20260925 02:50:38.251005 7012 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:38.253247 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd9c45cbacc42cd971371d05d41ce5f. Latest consensus state: current_term: 1 leader_uuid: "cdd9c45cbacc42cd971371d05d41ce5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd9c45cbacc42cd971371d05d41ce5f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34789 } } }
I20260925 02:50:38.253739 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:38.254988 7018 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdd9c45cbacc42cd971371d05d41ce5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd9c45cbacc42cd971371d05d41ce5f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34789 } } }
I20260925 02:50:38.255376 7018 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:38.256345 7023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:38.263418 7023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:38.274982 7023 catalog_manager.cc:1347] Generated new cluster ID: 6ff379b584994d2eb1669b01b4ede4c2
I20260925 02:50:38.275164 7023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:38.297420 7023 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:38.299175 7023 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:38.311820 7023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f: Generated new TSK 0
I20260925 02:50:38.313309 7023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:50:38.384775 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34789
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:38.796376 7038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:38.796739 7038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:38.796924 7038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:38.804512 7038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:38.804687 7038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:38.804884 7038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:50:38.815485 7038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:38.817148 7038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:38.818670 7038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:38.832610 7043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:38.832638 7044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:38.833216 7046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:38.833882 7038 server_base.cc:1034] running on GCE node
I20260925 02:50:38.834661 7038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:38.835748 7038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:38.836978 7038 hybrid_clock.cc:648] HybridClock initialized: now 1790304638836923 us; error 42 us; skew 500 ppm
I20260925 02:50:38.837334 7038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:38 dist-test-slave-g1xh krb5kdc[6931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304638, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:38.840900 7038 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:50:38.842272 7038 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:38.842378 7038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:38.842751 7038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:38.846079 7038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "256e39476e0f48af99cfdc44cb71bdd8"
format_stamp: "Formatted at 2026-09-25 02:50:38 on dist-test-slave-g1xh"
server_key: "6b210bd6ac7d5dd08c0bae3e3525685d"
server_key_iv: "168db487f410f491c5e8876184c47ac9"
server_key_version: "encryptionkey@0"
I20260925 02:50:38.847098 7038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "256e39476e0f48af99cfdc44cb71bdd8"
format_stamp: "Formatted at 2026-09-25 02:50:38 on dist-test-slave-g1xh"
server_key: "6b210bd6ac7d5dd08c0bae3e3525685d"
server_key_iv: "168db487f410f491c5e8876184c47ac9"
server_key_version: "encryptionkey@0"
I20260925 02:50:38.852850 7038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 02:50:38.856905 7052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:38.858507 7038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:50:38.858721 7038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "256e39476e0f48af99cfdc44cb71bdd8"
format_stamp: "Formatted at 2026-09-25 02:50:38 on dist-test-slave-g1xh"
server_key: "6b210bd6ac7d5dd08c0bae3e3525685d"
server_key_iv: "168db487f410f491c5e8876184c47ac9"
server_key_version: "encryptionkey@0"
I20260925 02:50:38.858974 7038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:38.872974 7038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:38.877413 7038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:38.877727 7038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:38.878768 7038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:38.880294 7038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:38.880427 7038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:38.880566 7038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:38.880658 7038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:38.916215 7038 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:43835
I20260925 02:50:38.916277 7166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:43835 every 8 connection(s)
I20260925 02:50:38.917146 7038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:38.926539 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 7038
I20260925 02:50:38.926771 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:38.927685 2737 external_mini_cluster.cc:1428] Setting key 410b21fc865777faa62184141f0f4277
Sep 25 02:50:38 dist-test-slave-g1xh krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304638, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:38.940061 7169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:38.919725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:40973 (local address 127.2.172.126:34789)
0925 02:50:38.920243 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:38.920250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:38.921279 (+ 1029us) server_negotiation.cc:407] Connection header received
0925 02:50:38.923153 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:38.923157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:38.923237 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:38.923347 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:38.925045 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.925915 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.928554 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.929126 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.929292 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:38.933838 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:38.933844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:38.933848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:38.933887 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:38.936720 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:38.937552 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:38.937565 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:38.937568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:38.937690 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:38.938143 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:38.938148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:38.938150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:38.938441 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:38.938605 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:38.939369 (+ 764us) server_negotiation.cc:299] Negotiation successful
0925 02:50:38.939662 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":235,"threads_started":1}
I20260925 02:50:38.941151 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:38.920386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:34789 (local address 127.2.172.65:40973)
0925 02:50:38.921102 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:38.921142 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:38.922809 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:38.923656 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:38.923666 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:38.924219 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:38.924849 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:38.924867 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.926129 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.926135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:38.928369 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:38.928389 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.929409 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.929419 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:38.929738 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:38.930504 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:38.930543 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:38.933514 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:38.936897 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:38.936908 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:38.936928 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:38.937406 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:38.937809 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:38.937814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:38.937818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:38.938009 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:38.938574 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:38.938583 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:38.938970 (+ 387us) client_negotiation.cc:769] Sending connection context
0925 02:50:38.939262 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 02:50:38.939908 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":279,"threads_started":1}
I20260925 02:50:38.942957 7167 heartbeater.cc:347] Connected to a master server at 127.2.172.126:34789
I20260925 02:50:38.943312 7167 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:38.944228 7167 heartbeater.cc:511] Master 127.2.172.126:34789 requested a full tablet report, sending...
I20260925 02:50:38.947022 6977 ts_manager.cc:194] Registered new tserver with Master: 256e39476e0f48af99cfdc44cb71bdd8 (127.2.172.65:43835)
I20260925 02:50:38.949357 6977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:40973
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:50:38.983851 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34789
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:39.340008 7173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:39.340397 7173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:39.340582 7173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:39.348872 7173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:39.348999 7173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:39.349217 7173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:50:39.360874 7173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:39.362327 7173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:39.364048 7173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:39.378964 7178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:39.378914 7179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:39.378942 7181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:39.379899 7173 server_base.cc:1034] running on GCE node
I20260925 02:50:39.380576 7173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:39.381672 7173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:39.382930 7173 hybrid_clock.cc:648] HybridClock initialized: now 1790304639382878 us; error 50 us; skew 500 ppm
I20260925 02:50:39.383339 7173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:39 dist-test-slave-g1xh krb5kdc[6931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304639, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:39.386809 7173 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:50:39.388284 7173 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:39.388428 7173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:39.388764 7173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:39.392155 7173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "a35a42665ef94413b5fdbf83ffd8eb3c"
format_stamp: "Formatted at 2026-09-25 02:50:39 on dist-test-slave-g1xh"
server_key: "58cc81773bf7abe8faaed1f844ed95d0"
server_key_iv: "e0c7fee2fd142541f1faf133c5cc1e1a"
server_key_version: "encryptionkey@0"
I20260925 02:50:39.393064 7173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "a35a42665ef94413b5fdbf83ffd8eb3c"
format_stamp: "Formatted at 2026-09-25 02:50:39 on dist-test-slave-g1xh"
server_key: "58cc81773bf7abe8faaed1f844ed95d0"
server_key_iv: "e0c7fee2fd142541f1faf133c5cc1e1a"
server_key_version: "encryptionkey@0"
I20260925 02:50:39.398583 7173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:50:39.402621 7187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:39.404106 7173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:50:39.404304 7173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "a35a42665ef94413b5fdbf83ffd8eb3c"
format_stamp: "Formatted at 2026-09-25 02:50:39 on dist-test-slave-g1xh"
server_key: "58cc81773bf7abe8faaed1f844ed95d0"
server_key_iv: "e0c7fee2fd142541f1faf133c5cc1e1a"
server_key_version: "encryptionkey@0"
I20260925 02:50:39.404537 7173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:39.433772 7173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:39.438100 7173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:39.438397 7173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:39.439375 7173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:39.440894 7173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:39.441011 7173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:39.441141 7173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:39.441228 7173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:39.477543 7173 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:44687
I20260925 02:50:39.477639 7301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:44687 every 8 connection(s)
I20260925 02:50:39.478469 7173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:39.487028 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 7173
I20260925 02:50:39.487234 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:39.487810 2737 external_mini_cluster.cc:1428] Setting key 72e6ab5d11dd81c2d084fbd26ec7bffa
Sep 25 02:50:39 dist-test-slave-g1xh krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304639, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:39.499955 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:39.481200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:40127 (local address 127.2.172.126:34789)
0925 02:50:39.481670 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:39.481675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:39.482695 (+ 1020us) server_negotiation.cc:407] Connection header received
0925 02:50:39.484625 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:39.484629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:39.484700 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:39.484799 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:39.486487 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:39.487291 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:39.489310 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:39.489820 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:39.489982 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:39.493875 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:39.493880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:39.493884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:39.493921 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:39.496676 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:39.497546 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:39.497559 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:39.497562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:39.497690 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:39.498142 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:39.498146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:39.498148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:39.498417 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:39.498575 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:39.499309 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 02:50:39.499538 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":235,"threads_started":1}
I20260925 02:50:39.500973 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:39.481789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:34789 (local address 127.2.172.66:40127)
0925 02:50:39.482515 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:39.482561 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:39.484329 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:39.485034 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:39.485043 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:39.485655 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:39.486257 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:39.486276 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:39.487514 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:39.487518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:39.489143 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:39.489157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:39.490095 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:39.490103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:39.490351 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:39.491081 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:39.491110 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:39.493562 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:39.496861 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:39.496870 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:39.496896 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:39.497359 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:39.497812 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:39.497817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:39.497821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:39.498013 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:39.498548 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:39.498556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:39.498921 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 02:50:39.499198 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 02:50:39.499783 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":292,"threads_started":1}
I20260925 02:50:39.502702 7302 heartbeater.cc:347] Connected to a master server at 127.2.172.126:34789
I20260925 02:50:39.503072 7302 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:39.503932 7302 heartbeater.cc:511] Master 127.2.172.126:34789 requested a full tablet report, sending...
I20260925 02:50:39.505867 6977 ts_manager.cc:194] Registered new tserver with Master: a35a42665ef94413b5fdbf83ffd8eb3c (127.2.172.66:44687)
I20260925 02:50:39.507151 6977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:40127
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:50:39.539682 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34789
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:39.878252 7308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:39.878547 7308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:39.878645 7308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:39.885996 7308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:39.886116 7308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:39.886235 7308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:50:39.896243 7308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:39.897563 7308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:39.898816 7308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:39.912652 7313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:39.912755 7314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:39.913313 7308 server_base.cc:1034] running on GCE node
W20260925 02:50:39.913511 7316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:39.914186 7308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:39.915163 7308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:39.916396 7308 hybrid_clock.cc:648] HybridClock initialized: now 1790304639916338 us; error 46 us; skew 500 ppm
I20260925 02:50:39.916715 7308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:39 dist-test-slave-g1xh krb5kdc[6931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304639, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:39.919893 7308 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:50:39.921103 7308 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:39.921190 7308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:39.921567 7308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:39.924471 7308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "69f524e2487a42958e0f9865f72a779e"
format_stamp: "Formatted at 2026-09-25 02:50:39 on dist-test-slave-g1xh"
server_key: "3793b9dd2431f9b61031ab98275b5b41"
server_key_iv: "01204bb5b7b74f5ec021dc2dc11252f4"
server_key_version: "encryptionkey@0"
I20260925 02:50:39.925275 7308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "69f524e2487a42958e0f9865f72a779e"
format_stamp: "Formatted at 2026-09-25 02:50:39 on dist-test-slave-g1xh"
server_key: "3793b9dd2431f9b61031ab98275b5b41"
server_key_iv: "01204bb5b7b74f5ec021dc2dc11252f4"
server_key_version: "encryptionkey@0"
I20260925 02:50:39.930296 7308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:50:39.933843 7322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:39.935175 7308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:50:39.935348 7308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "69f524e2487a42958e0f9865f72a779e"
format_stamp: "Formatted at 2026-09-25 02:50:39 on dist-test-slave-g1xh"
server_key: "3793b9dd2431f9b61031ab98275b5b41"
server_key_iv: "01204bb5b7b74f5ec021dc2dc11252f4"
server_key_version: "encryptionkey@0"
I20260925 02:50:39.935607 7308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:39.950335 7308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:39.952989 7167 heartbeater.cc:503] Master 127.2.172.126:34789 was elected leader, sending a full tablet report...
I20260925 02:50:39.955432 7308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:39.955763 7308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:39.956867 7308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:39.958307 7308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:39.958393 7308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:39.958464 7308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:39.958521 7308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:39.991369 7308 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:32891
I20260925 02:50:39.991421 7436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:32891 every 8 connection(s)
I20260925 02:50:39.992297 7308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:39.993305 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 7308
I20260925 02:50:39.993511 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:39.993961 2737 external_mini_cluster.cc:1428] Setting key 1db993f70e1bd39c3a1b81b20d71716b
Sep 25 02:50:40 dist-test-slave-g1xh krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304639, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:40.014333 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:39.995280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:42871 (local address 127.2.172.126:34789)
0925 02:50:39.995483 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:39.995489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:39.997066 (+ 1577us) server_negotiation.cc:407] Connection header received
0925 02:50:39.999152 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:39.999156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:39.999247 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:39.999349 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:40.001157 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.001992 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.003875 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.004294 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.004398 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.008194 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:40.008199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:40.008202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:40.008237 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:40.011329 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:40.012237 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:40.012241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:40.012244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:40.012380 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:40.012754 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:40.012757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:40.012759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:40.013052 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:40.013172 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:40.013912 (+ 740us) server_negotiation.cc:299] Negotiation successful
0925 02:50:40.014087 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:50:40.015336 7439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:39.995903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:34789 (local address 127.2.172.67:42871)
0925 02:50:39.996884 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:39.996935 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:39.998845 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:39.999720 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:39.999731 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:40.000377 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:40.000978 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:40.000996 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.002178 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.002183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:40.003732 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:40.003747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.004519 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.004525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:40.004722 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.005514 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:40.005548 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:40.007906 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:40.011489 (+ 3583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:40.011497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:40.011511 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:40.012085 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:40.012482 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:40.012485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:40.012487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:40.012654 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:40.013171 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:40.013176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:40.013558 (+ 382us) client_negotiation.cc:769] Sending connection context
0925 02:50:40.013790 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 02:50:40.014210 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":420,"threads_started":1}
I20260925 02:50:40.016845 7437 heartbeater.cc:347] Connected to a master server at 127.2.172.126:34789
I20260925 02:50:40.017127 7437 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:40.017769 7437 heartbeater.cc:511] Master 127.2.172.126:34789 requested a full tablet report, sending...
I20260925 02:50:40.019230 6977 ts_manager.cc:194] Registered new tserver with Master: 69f524e2487a42958e0f9865f72a779e (127.2.172.67:32891)
I20260925 02:50:40.020130 6977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:42871
I20260925 02:50:40.022902 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:50:40.040364 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.027485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.2.172.126:34789)
0925 02:50:40.027729 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:40.027737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:40.028229 (+ 492us) server_negotiation.cc:407] Connection header received
0925 02:50:40.028380 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:40.028384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:40.028444 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:40.028526 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:40.030090 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.030779 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.032105 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.032536 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.032635 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.034915 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:40.034921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:40.034924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:40.034956 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:40.037625 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:40.038326 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:40.038330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:40.038333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:40.038475 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:40.038824 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:40.038827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:40.038829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:40.039182 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:40.039392 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:40.039748 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 02:50:40.039991 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 02:50:40.044304 6977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:40.051594 6977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:40.084707 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.063926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:44687 (local address 127.0.0.1:43706)
0925 02:50:40.074290 (+ 10364us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:40.074348 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:40.075679 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:40.076814 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:40.076826 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:40.076863 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:40.077384 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:40.077399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.079234 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.079240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:40.081922 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:40.081938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.083616 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.083623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:40.083824 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.083876 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 02:50:40.084022 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 02:50:40.084224 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10003,"mutex_wait_us":1,"thread_start_us":13368,"threads_started":1}
I20260925 02:50:40.085525 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.064192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.2.172.66:44687)
0925 02:50:40.064804 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:40.064815 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:40.074529 (+ 9714us) server_negotiation.cc:407] Connection header received
0925 02:50:40.076350 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:40.076357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:40.076562 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:40.076817 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:40.077595 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.079086 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.082087 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.083051 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.083219 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.083458 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:40.084987 (+ 1529us) server_negotiation.cc:299] Negotiation successful
0925 02:50:40.085180 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":777,"threads_started":1}
I20260925 02:50:40.086119 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.065626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.2.172.67:32891)
0925 02:50:40.075036 (+ 9410us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:40.075049 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:40.077492 (+ 2443us) server_negotiation.cc:407] Connection header received
0925 02:50:40.077777 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:40.077785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:40.077999 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:40.078152 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:40.079165 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.080308 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.083190 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.084685 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.084829 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.085059 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:40.085625 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 02:50:40.085826 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9204,"thread_start_us":8841,"threads_started":1}
I20260925 02:50:40.086701 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.065548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.2.172.65:43835)
0925 02:50:40.076096 (+ 10548us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:40.076103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:40.077151 (+ 1048us) server_negotiation.cc:407] Connection header received
0925 02:50:40.077370 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:40.077379 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:40.077597 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:40.077811 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:40.078647 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.080332 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.083699 (+ 3367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.085204 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.085335 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.085561 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:40.086187 (+ 626us) server_negotiation.cc:299] Negotiation successful
0925 02:50:40.086353 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10398,"thread_start_us":10113,"threads_started":1}
I20260925 02:50:40.086942 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.065240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:32891 (local address 127.0.0.1:54750)
0925 02:50:40.077378 (+ 12138us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:40.077394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:40.077593 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:40.078474 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:40.078481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:40.078512 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:40.078910 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:40.078934 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.080457 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.080464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:40.083015 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:40.083030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.085191 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.085197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:40.085392 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.085434 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:50:40.086307 (+ 873us) client_negotiation.cc:240] Negotiation successful
0925 02:50:40.086461 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12002,"mutex_wait_us":1,"thread_start_us":11478,"threads_started":1}
I20260925 02:50:40.087240 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.064643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43835 (local address 127.0.0.1:32880)
0925 02:50:40.077010 (+ 12367us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:40.077033 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:40.077176 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:40.078050 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:40.078055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:40.078080 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:40.078461 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:40.078473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.080831 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.080836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:40.083494 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:40.083504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.085868 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.085873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:40.085996 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.086018 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:50:40.086131 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:50:40.086430 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12219,"thread_start_us":9113,"threads_started":1}
I20260925 02:50:40.089991 7235 tablet_service.cc:1467] Processing CreateTablet for tablet 0f7a8050c69e4c84b42360572bef735b (DEFAULT_TABLE table=test-table [id=e5a8c0abf9424804870760e42cc72dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:40.090343 7370 tablet_service.cc:1467] Processing CreateTablet for tablet 0f7a8050c69e4c84b42360572bef735b (DEFAULT_TABLE table=test-table [id=e5a8c0abf9424804870760e42cc72dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:40.090796 7101 tablet_service.cc:1467] Processing CreateTablet for tablet 0f7a8050c69e4c84b42360572bef735b (DEFAULT_TABLE table=test-table [id=e5a8c0abf9424804870760e42cc72dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:40.091675 7235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f7a8050c69e4c84b42360572bef735b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:40.091686 7370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f7a8050c69e4c84b42360572bef735b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:40.092101 7101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f7a8050c69e4c84b42360572bef735b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:40.103909 7455 tablet_bootstrap.cc:492] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e: Bootstrap starting.
I20260925 02:50:40.106524 7456 tablet_bootstrap.cc:492] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c: Bootstrap starting.
I20260925 02:50:40.109055 7455 tablet_bootstrap.cc:654] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:40.110247 7456 tablet_bootstrap.cc:654] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:40.110430 7455 log.cc:824] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:40.111579 7456 log.cc:824] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:40.114358 7455 tablet_bootstrap.cc:492] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e: No bootstrap required, opened a new log
I20260925 02:50:40.114730 7455 ts_tablet_manager.cc:1397] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260925 02:50:40.112015 7457 tablet_bootstrap.cc:492] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8: Bootstrap starting.
I20260925 02:50:40.117594 7456 tablet_bootstrap.cc:492] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c: No bootstrap required, opened a new log
I20260925 02:50:40.117887 7456 ts_tablet_manager.cc:1397] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260925 02:50:40.118386 7457 tablet_bootstrap.cc:654] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:40.119722 7457 log.cc:824] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:40.122874 7457 tablet_bootstrap.cc:492] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8: No bootstrap required, opened a new log
I20260925 02:50:40.123159 7457 ts_tablet_manager.cc:1397] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260925 02:50:40.125464 7455 raft_consensus.cc:348] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 44687 } } peers { permanent_uuid: "256e39476e0f48af99cfdc44cb71bdd8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43835 } } peers { permanent_uuid: "69f524e2487a42958e0f9865f72a779e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 32891 } }
I20260925 02:50:40.125965 7455 raft_consensus.cc:374] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:40.126071 7455 raft_consensus.cc:729] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f524e2487a42958e0f9865f72a779e, State: Initialized, Role: FOLLOWER
I20260925 02:50:40.126482 7456 raft_consensus.cc:348] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 44687 } } peers { permanent_uuid: "256e39476e0f48af99cfdc44cb71bdd8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43835 } } peers { permanent_uuid: "69f524e2487a42958e0f9865f72a779e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 32891 } }
I20260925 02:50:40.126837 7456 raft_consensus.cc:374] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:40.126726 7455 consensus_queue.cc:260] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 44687 } } peers { permanent_uuid: "256e39476e0f48af99cfdc44cb71bdd8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43835 } } peers { permanent_uuid: "69f524e2487a42958e0f9865f72a779e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 32891 } }
I20260925 02:50:40.126957 7456 raft_consensus.cc:729] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35a42665ef94413b5fdbf83ffd8eb3c, State: Initialized, Role: FOLLOWER
I20260925 02:50:40.127521 7456 consensus_queue.cc:260] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 44687 } } peers { permanent_uuid: "256e39476e0f48af99cfdc44cb71bdd8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43835 } } peers { permanent_uuid: "69f524e2487a42958e0f9865f72a779e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 32891 } }
I20260925 02:50:40.128720 7302 heartbeater.cc:503] Master 127.2.172.126:34789 was elected leader, sending a full tablet report...
I20260925 02:50:40.129287 7456 ts_tablet_manager.cc:1428] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 02:50:40.130173 7437 heartbeater.cc:503] Master 127.2.172.126:34789 was elected leader, sending a full tablet report...
I20260925 02:50:40.131119 7455 ts_tablet_manager.cc:1428] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260925 02:50:40.133759 7457 raft_consensus.cc:348] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 44687 } } peers { permanent_uuid: "256e39476e0f48af99cfdc44cb71bdd8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43835 } } peers { permanent_uuid: "69f524e2487a42958e0f9865f72a779e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 32891 } }
I20260925 02:50:40.134160 7457 raft_consensus.cc:374] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:40.134267 7457 raft_consensus.cc:729] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256e39476e0f48af99cfdc44cb71bdd8, State: Initialized, Role: FOLLOWER
I20260925 02:50:40.135023 7457 consensus_queue.cc:260] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 44687 } } peers { permanent_uuid: "256e39476e0f48af99cfdc44cb71bdd8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43835 } } peers { permanent_uuid: "69f524e2487a42958e0f9865f72a779e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 32891 } }
I20260925 02:50:40.136776 7457 ts_tablet_manager.cc:1428] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
W20260925 02:50:40.169448 7168 tablet.cc:2369] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:50:40.230479 7303 tablet.cc:2369] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:50:40.243960 7438 tablet.cc:2369] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:40.323843 7463 raft_consensus.cc:482] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:40.324227 7463 raft_consensus.cc:504] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 44687 } } peers { permanent_uuid: "256e39476e0f48af99cfdc44cb71bdd8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43835 } } peers { permanent_uuid: "69f524e2487a42958e0f9865f72a779e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 32891 } }
I20260925 02:50:40.326088 7463 leader_election.cc:288] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a35a42665ef94413b5fdbf83ffd8eb3c (127.2.172.66:44687), 69f524e2487a42958e0f9865f72a779e (127.2.172.67:32891)
I20260925 02:50:40.332901 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.326541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:55649 (local address 127.2.172.67:32891)
0925 02:50:40.326749 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:40.326755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:40.327408 (+ 653us) server_negotiation.cc:407] Connection header received
0925 02:50:40.327612 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:40.327618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:40.327691 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:40.327789 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:40.328360 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.329131 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.331066 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.331937 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.332126 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.332207 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:40.332503 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 02:50:40.332627 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:50:40.333129 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.326588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:58779 (local address 127.2.172.66:44687)
0925 02:50:40.326763 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:40.326771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:40.327887 (+ 1116us) server_negotiation.cc:407] Connection header received
0925 02:50:40.328139 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:40.328149 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:40.328210 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:40.328327 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:40.328816 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.329721 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.331427 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.332201 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.332392 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.332476 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:40.332746 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 02:50:40.332874 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:50:40.333283 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.326576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:44687 (local address 127.2.172.65:58779)
0925 02:50:40.327767 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:40.327782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:40.327984 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:40.328341 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:40.328345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:40.328369 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:40.328658 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:40.328666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.329864 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.329879 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:40.331259 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:40.331270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.332425 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.332432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:40.332588 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.332610 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:50:40.332718 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 02:50:40.332854 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"spinlock_wait_cycles":68224,"thread_start_us":861,"threads_started":1}
I20260925 02:50:40.333307 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.326547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:32891 (local address 127.2.172.65:55649)
0925 02:50:40.327232 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:40.327248 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:40.327426 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:40.327842 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:40.327847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:40.327871 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:40.328207 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:40.328215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.329295 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.329303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:40.330904 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:40.330915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.332158 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.332171 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:40.332338 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.332365 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:50:40.332478 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:50:40.332612 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":1,"thread_start_us":420,"threads_started":1}
I20260925 02:50:40.334277 7390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a8050c69e4c84b42360572bef735b" candidate_uuid: "256e39476e0f48af99cfdc44cb71bdd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f524e2487a42958e0f9865f72a779e" is_pre_election: true
I20260925 02:50:40.334276 7257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a8050c69e4c84b42360572bef735b" candidate_uuid: "256e39476e0f48af99cfdc44cb71bdd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c" is_pre_election: true
I20260925 02:50:40.334780 7390 raft_consensus.cc:2454] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 256e39476e0f48af99cfdc44cb71bdd8 in term 0.
I20260925 02:50:40.334792 7257 raft_consensus.cc:2454] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 256e39476e0f48af99cfdc44cb71bdd8 in term 0.
I20260925 02:50:40.335489 7056 leader_election.cc:302] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 256e39476e0f48af99cfdc44cb71bdd8, a35a42665ef94413b5fdbf83ffd8eb3c; no voters:
I20260925 02:50:40.336030 7463 raft_consensus.cc:2781] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:40.336207 7463 raft_consensus.cc:482] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:40.336344 7463 raft_consensus.cc:3037] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:40.338656 7463 raft_consensus.cc:504] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 44687 } } peers { permanent_uuid: "256e39476e0f48af99cfdc44cb71bdd8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43835 } } peers { permanent_uuid: "69f524e2487a42958e0f9865f72a779e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 32891 } }
I20260925 02:50:40.339327 7463 leader_election.cc:288] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [CANDIDATE]: Term 1 election: Requested vote from peers a35a42665ef94413b5fdbf83ffd8eb3c (127.2.172.66:44687), 69f524e2487a42958e0f9865f72a779e (127.2.172.67:32891)
I20260925 02:50:40.339924 7257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a8050c69e4c84b42360572bef735b" candidate_uuid: "256e39476e0f48af99cfdc44cb71bdd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c"
I20260925 02:50:40.340019 7390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a8050c69e4c84b42360572bef735b" candidate_uuid: "256e39476e0f48af99cfdc44cb71bdd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f524e2487a42958e0f9865f72a779e"
I20260925 02:50:40.340199 7257 raft_consensus.cc:3037] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:40.340240 7390 raft_consensus.cc:3037] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:40.343569 7257 raft_consensus.cc:2454] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 256e39476e0f48af99cfdc44cb71bdd8 in term 1.
I20260925 02:50:40.343576 7390 raft_consensus.cc:2454] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 256e39476e0f48af99cfdc44cb71bdd8 in term 1.
I20260925 02:50:40.344091 7056 leader_election.cc:302] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 256e39476e0f48af99cfdc44cb71bdd8, a35a42665ef94413b5fdbf83ffd8eb3c; no voters:
I20260925 02:50:40.344463 7463 raft_consensus.cc:2781] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:40.344987 7463 raft_consensus.cc:686] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 1 LEADER]: Becoming Leader. State: Replica: 256e39476e0f48af99cfdc44cb71bdd8, State: Running, Role: LEADER
I20260925 02:50:40.345518 7463 consensus_queue.cc:237] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 44687 } } peers { permanent_uuid: "256e39476e0f48af99cfdc44cb71bdd8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43835 } } peers { permanent_uuid: "69f524e2487a42958e0f9865f72a779e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 32891 } }
I20260925 02:50:40.350595 6977 catalog_manager.cc:5495] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 256e39476e0f48af99cfdc44cb71bdd8 (127.2.172.65). New cstate: current_term: 1 leader_uuid: "256e39476e0f48af99cfdc44cb71bdd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "256e39476e0f48af99cfdc44cb71bdd8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f524e2487a42958e0f9865f72a779e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 32891 } health_report { overall_health: UNKNOWN } } }
I20260925 02:50:40.394007 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.388060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.2.172.65:43835)
0925 02:50:40.388322 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:40.388330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:40.388433 (+ 103us) server_negotiation.cc:407] Connection header received
0925 02:50:40.388705 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:40.388709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:40.388810 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:40.388983 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:40.389437 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.390153 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.391660 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.392134 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.392279 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.392817 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:40.393362 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:40.393524 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:40.393638 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 02:50:40.393723 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 02:50:40.402647 7390 raft_consensus.cc:1261] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e [term 1 FOLLOWER]: Refusing update from remote peer 256e39476e0f48af99cfdc44cb71bdd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:40.402665 7257 raft_consensus.cc:1261] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c [term 1 FOLLOWER]: Refusing update from remote peer 256e39476e0f48af99cfdc44cb71bdd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:40.403781 7466 consensus_queue.cc:1035] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35a42665ef94413b5fdbf83ffd8eb3c" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:40.404166 7463 consensus_queue.cc:1035] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f524e2487a42958e0f9865f72a779e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 32891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:40.415539 7469 mvcc.cc:205] Tried to move back new op lower bound from 7333087807078879232 to 7333087806858805248. Current Snapshot: MvccSnapshot[applied={T|T < 7333087807078879232}]
I20260925 02:50:40.430928 6977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:40.431234 6977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:40.435482 6977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd9c45cbacc42cd971371d05d41ce5f: Sending DeleteTablet for 3 replicas of tablet 0f7a8050c69e4c84b42360572bef735b
I20260925 02:50:40.436779 7101 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f7a8050c69e4c84b42360572bef735b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:40 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:32880
I20260925 02:50:40.436779 7235 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f7a8050c69e4c84b42360572bef735b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:40 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:43706
I20260925 02:50:40.436779 7370 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f7a8050c69e4c84b42360572bef735b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:40 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:54750
I20260925 02:50:40.441543 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 7038
I20260925 02:50:40.452569 7480 tablet_replica.cc:330] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8: stopping tablet replica
I20260925 02:50:40.453125 7480 raft_consensus.cc:2229] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:50:40.453610 7479 tablet_replica.cc:330] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e: stopping tablet replica
I20260925 02:50:40.453896 7480 raft_consensus.cc:2258] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:40.454118 7479 raft_consensus.cc:2229] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:40.454668 7479 raft_consensus.cc:2258] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:40.456393 7480 ts_tablet_manager.cc:1905] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:40.457286 7478 tablet_replica.cc:330] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c: stopping tablet replica
I20260925 02:50:40.457804 7478 raft_consensus.cc:2229] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:40.458199 7479 ts_tablet_manager.cc:1905] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:40.463436 7478 raft_consensus.cc:2258] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:40.464622 7480 ts_tablet_manager.cc:1918] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:40.464829 7480 log.cc:1196] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/0f7a8050c69e4c84b42360572bef735b
I20260925 02:50:40.465447 7480 ts_tablet_manager.cc:1939] T 0f7a8050c69e4c84b42360572bef735b P 256e39476e0f48af99cfdc44cb71bdd8: Deleting consensus metadata
I20260925 02:50:40.465627 7478 ts_tablet_manager.cc:1905] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:40.467834 6961 catalog_manager.cc:4841] TS 256e39476e0f48af99cfdc44cb71bdd8 (127.2.172.65:43835): tablet 0f7a8050c69e4c84b42360572bef735b (table test-table [id=e5a8c0abf9424804870760e42cc72dee]) successfully deleted
I20260925 02:50:40.468755 7479 ts_tablet_manager.cc:1918] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:40.468945 7479 log.cc:1196] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/0f7a8050c69e4c84b42360572bef735b
I20260925 02:50:40.469527 7479 ts_tablet_manager.cc:1939] T 0f7a8050c69e4c84b42360572bef735b P 69f524e2487a42958e0f9865f72a779e: Deleting consensus metadata
I20260925 02:50:40.471354 6962 catalog_manager.cc:4841] TS 69f524e2487a42958e0f9865f72a779e (127.2.172.67:32891): tablet 0f7a8050c69e4c84b42360572bef735b (table test-table [id=e5a8c0abf9424804870760e42cc72dee]) successfully deleted
I20260925 02:50:40.472514 7478 ts_tablet_manager.cc:1918] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:40.472709 7478 log.cc:1196] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/0f7a8050c69e4c84b42360572bef735b
I20260925 02:50:40.473183 7478 ts_tablet_manager.cc:1939] T 0f7a8050c69e4c84b42360572bef735b P a35a42665ef94413b5fdbf83ffd8eb3c: Deleting consensus metadata
Sep 25 02:50:40 dist-test-slave-g1xh krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:40.474890 6963 catalog_manager.cc:4841] TS a35a42665ef94413b5fdbf83ffd8eb3c (127.2.172.66:44687): tablet 0f7a8050c69e4c84b42360572bef735b (table test-table [id=e5a8c0abf9424804870760e42cc72dee]) successfully deleted
I20260925 02:50:40.484045 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.442679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.2.172.65:43835)
0925 02:50:40.442872 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:40.442879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:40.458814 (+ 15935us) server_negotiation.cc:407] Connection header received
0925 02:50:40.459328 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:40.459334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:40.459448 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:40.459530 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:40.461448 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.462866 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.464846 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.465562 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.465673 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.475978 (+ 10305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:40.475994 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:40.476009 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:40.476049 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:40.480362 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:40.481494 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:40.481505 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:40.481509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:40.481700 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:40.482189 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:40.482197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:40.482203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:40.482660 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:40.482964 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:40.483363 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 02:50:40.483645 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:50:40.484941 7162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:40.594447 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 7038
I20260925 02:50:40.614341 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 7173
Sep 25 02:50:40 dist-test-slave-g1xh krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:50:40.631341 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.615313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.2.172.66:44687)
0925 02:50:40.615543 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:40.615594 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:40.615616 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:50:40.615841 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:40.615849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:40.615927 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:40.616045 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:40.617455 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.618285 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.619804 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.620207 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.620355 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.624860 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:40.624879 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:40.624895 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:40.624945 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:40.628253 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:40.629064 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:40.629072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:40.629074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:40.629222 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:40.629675 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:40.629681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:40.629683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:40.630185 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:40.630419 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:40.630769 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 02:50:40.630950 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 02:50:40.632190 7297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:40.732774 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 7173
I20260925 02:50:40.751503 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 7308
Sep 25 02:50:40 dist-test-slave-g1xh krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:50:40.768443 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.752622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.2.172.67:32891)
0925 02:50:40.752814 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:40.752819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:40.752838 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:50:40.752984 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:40.752988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:40.753071 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:40.753159 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:40.754731 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.755978 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.757466 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:40.757971 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:40.758116 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:40.762580 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:40.762595 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:40.762612 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:40.762647 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:40.765774 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:40.766427 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:40.766435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:40.766437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:40.766573 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:40.766948 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:40.766952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:40.766954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:40.767332 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:40.767638 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:40.768004 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 02:50:40.768178 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:50:40.769143 7432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:40.869040 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 7308
I20260925 02:50:40.888094 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 6944
I20260925 02:50:40.889344 7007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:41.001396 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 6944
2026-09-25T02:50:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4693 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T02:50:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:41Z Disabled control of system clock
I20260925 02:50:41.073478 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:45759
--webserver_interface=127.2.172.126
--webserver_port=0
--builtin_ntp_servers=127.2.172.84:37309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:45759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:50:41.409135 7489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:41.409457 7489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:41.409544 7489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:41.416975 7489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:41.417096 7489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:41.417135 7489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:41.417184 7489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:41.417222 7489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:41.426668 7489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:45759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:45759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:41.427982 7489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:41.429690 7489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:41.442206 7495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:41.442220 7494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:41.442498 7497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:41.443192 7489 server_base.cc:1034] running on GCE node
I20260925 02:50:41.444181 7489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:41.445930 7489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:41.447209 7489 hybrid_clock.cc:648] HybridClock initialized: now 1790304641447129 us; error 75 us; skew 500 ppm
I20260925 02:50:41.447659 7489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:50:41.450222 7489 webserver.cc:460] Webserver started at http://127.2.172.126:42237/ using document root <none> and password file <none>
I20260925 02:50:41.450963 7489 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:41.451102 7489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:41.451444 7489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:41.454615 7489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "90fe37d6e7be44ae9d9e8f07241fcb7b"
format_stamp: "Formatted at 2026-09-25 02:50:41 on dist-test-slave-g1xh"
server_key: "8c0e8a2f4149e11f6ed1b120ade0109a"
server_key_iv: "8e6a73cd25c3f5aacaedf8df2dbfc309"
server_key_version: "encryptionkey@0"
I20260925 02:50:41.455433 7489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "90fe37d6e7be44ae9d9e8f07241fcb7b"
format_stamp: "Formatted at 2026-09-25 02:50:41 on dist-test-slave-g1xh"
server_key: "8c0e8a2f4149e11f6ed1b120ade0109a"
server_key_iv: "8e6a73cd25c3f5aacaedf8df2dbfc309"
server_key_version: "encryptionkey@0"
I20260925 02:50:41.461396 7489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:50:41.465421 7503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:41.467103 7489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:50:41.467408 7489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "90fe37d6e7be44ae9d9e8f07241fcb7b"
format_stamp: "Formatted at 2026-09-25 02:50:41 on dist-test-slave-g1xh"
server_key: "8c0e8a2f4149e11f6ed1b120ade0109a"
server_key_iv: "8e6a73cd25c3f5aacaedf8df2dbfc309"
server_key_version: "encryptionkey@0"
I20260925 02:50:41.467746 7489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:41.474284 7489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:41.475184 7489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:41.475644 7489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:41.496723 7489 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:45759
I20260925 02:50:41.496742 7554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:45759 every 8 connection(s)
I20260925 02:50:41.498287 7489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:41.502842 7555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:41.504720 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 7489
I20260925 02:50:41.504974 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:41.505919 2737 external_mini_cluster.cc:1428] Setting key a624a0056b63cb3544fb9b0a87ca3ab0
I20260925 02:50:41.518982 7555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b: Bootstrap starting.
I20260925 02:50:41.520620 7558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:41.508395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.2.172.126:45759)
0925 02:50:41.509570 (+ 1175us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:41.509582 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:41.509623 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:50:41.511218 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:41.511242 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:41.511776 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:41.512171 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:41.518274 (+ 6103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:41.518288 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:41.518309 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:41.518334 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:41.518442 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:41.518476 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:41.518746 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:41.518892 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 02:50:41.519212 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":475,"threads_started":1}
I20260925 02:50:41.523046 7555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:41.524371 7555 log.cc:824] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:41.527664 7555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b: No bootstrap required, opened a new log
I20260925 02:50:41.535991 7555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fe37d6e7be44ae9d9e8f07241fcb7b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45759 } }
I20260925 02:50:41.536521 7555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:41.536659 7555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90fe37d6e7be44ae9d9e8f07241fcb7b, State: Initialized, Role: FOLLOWER
I20260925 02:50:41.537428 7555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fe37d6e7be44ae9d9e8f07241fcb7b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45759 } }
I20260925 02:50:41.537679 7555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:41.537791 7555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:41.537927 7555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:41.540460 7555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fe37d6e7be44ae9d9e8f07241fcb7b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45759 } }
I20260925 02:50:41.540966 7555 leader_election.cc:302] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90fe37d6e7be44ae9d9e8f07241fcb7b; no voters:
I20260925 02:50:41.541600 7555 leader_election.cc:288] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:41.541786 7561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:41.543902 7561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [term 1 LEADER]: Becoming Leader. State: Replica: 90fe37d6e7be44ae9d9e8f07241fcb7b, State: Running, Role: LEADER
I20260925 02:50:41.544250 7555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:41.544519 7561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fe37d6e7be44ae9d9e8f07241fcb7b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45759 } }
I20260925 02:50:41.547876 7563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90fe37d6e7be44ae9d9e8f07241fcb7b. Latest consensus state: current_term: 1 leader_uuid: "90fe37d6e7be44ae9d9e8f07241fcb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fe37d6e7be44ae9d9e8f07241fcb7b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45759 } } }
I20260925 02:50:41.548305 7563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:41.548926 7562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90fe37d6e7be44ae9d9e8f07241fcb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fe37d6e7be44ae9d9e8f07241fcb7b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45759 } } }
I20260925 02:50:41.549274 7562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:41.549551 7570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:41.558291 7570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:41.568351 7570 catalog_manager.cc:1347] Generated new cluster ID: 615c4d96be4640fba3c3325f5b491889
I20260925 02:50:41.568532 7570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:41.583721 7570 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:41.584236 7570 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:41.596988 7570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b: Generated new TSK 0
I20260925 02:50:41.598146 7570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:50:41.609279 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:45759
--builtin_ntp_servers=127.2.172.84:37309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:50:41.971468 7580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:41.971866 7580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:41.972043 7580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:41.979929 7580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:41.980072 7580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:41.980265 7580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:50:41.989850 7580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:45759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:41.991303 7580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:41.993175 7580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:42.006659 7586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:42.006693 7587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:42.006625 7589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:42.007880 7580 server_base.cc:1034] running on GCE node
I20260925 02:50:42.008584 7580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:42.009582 7580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:42.010828 7580 hybrid_clock.cc:648] HybridClock initialized: now 1790304642010782 us; error 59 us; skew 500 ppm
I20260925 02:50:42.011191 7580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:50:42.013748 7580 webserver.cc:460] Webserver started at http://127.2.172.65:36201/ using document root <none> and password file <none>
I20260925 02:50:42.014585 7580 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:42.014725 7580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:42.015064 7580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:42.018151 7580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "d85791cfac2143e1af841ff0dabf6506"
format_stamp: "Formatted at 2026-09-25 02:50:42 on dist-test-slave-g1xh"
server_key: "1f27e9feaa7ad82d9f56fc506c95ea85"
server_key_iv: "fbfda1118ddb57a6cddfe0bed613b535"
server_key_version: "encryptionkey@0"
I20260925 02:50:42.018988 7580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "d85791cfac2143e1af841ff0dabf6506"
format_stamp: "Formatted at 2026-09-25 02:50:42 on dist-test-slave-g1xh"
server_key: "1f27e9feaa7ad82d9f56fc506c95ea85"
server_key_iv: "fbfda1118ddb57a6cddfe0bed613b535"
server_key_version: "encryptionkey@0"
I20260925 02:50:42.024679 7580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 02:50:42.028687 7595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:42.030179 7580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:50:42.030385 7580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "d85791cfac2143e1af841ff0dabf6506"
format_stamp: "Formatted at 2026-09-25 02:50:42 on dist-test-slave-g1xh"
server_key: "1f27e9feaa7ad82d9f56fc506c95ea85"
server_key_iv: "fbfda1118ddb57a6cddfe0bed613b535"
server_key_version: "encryptionkey@0"
I20260925 02:50:42.030678 7580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:42.036629 7580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:42.037464 7580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:42.037738 7580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:42.038710 7580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:42.040319 7580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:42.040455 7580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:42.040582 7580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:42.040673 7580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:42.074671 7580 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:43983
I20260925 02:50:42.074769 7708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:43983 every 8 connection(s)
I20260925 02:50:42.076095 7580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:42.082020 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 7580
I20260925 02:50:42.082304 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:42.082939 2737 external_mini_cluster.cc:1428] Setting key 350dc3d48050f207b57cd67a46bfc0af
I20260925 02:50:42.084959 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:42.078836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:47251 (local address 127.2.172.126:45759)
0925 02:50:42.079267 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:42.079272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:42.080604 (+ 1332us) server_negotiation.cc:407] Connection header received
0925 02:50:42.082426 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:42.082432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:42.082521 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:42.082646 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:42.084115 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:42.084121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:42.084125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:42.084140 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:42.084181 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:42.084194 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:42.084354 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:42.084595 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 02:50:42.084646 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260925 02:50:42.085969 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:45759
--builtin_ntp_servers=127.2.172.84:37309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 02:50:42.086035 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:42.079453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:45759 (local address 127.2.172.65:47251)
0925 02:50:42.080442 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:42.080488 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:42.082082 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:42.082886 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:42.082894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:42.082996 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:42.083774 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:50:42.083777 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:42.083823 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:50:42.083918 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:42.084358 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:42.084370 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:42.084379 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:50:42.084674 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 02:50:42.084980 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":276,"threads_started":1}
I20260925 02:50:42.087602 7709 heartbeater.cc:347] Connected to a master server at 127.2.172.126:45759
I20260925 02:50:42.087941 7709 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:42.088696 7709 heartbeater.cc:511] Master 127.2.172.126:45759 requested a full tablet report, sending...
I20260925 02:50:42.091249 7520 ts_manager.cc:194] Registered new tserver with Master: d85791cfac2143e1af841ff0dabf6506 (127.2.172.65:43983)
W20260925 02:50:42.419651 7713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:42.419956 7713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:42.420130 7713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:42.427603 7713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:42.427721 7713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:42.427927 7713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:50:42.437566 7713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:45759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:42.438910 7713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:42.440750 7713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:42.455463 7721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:42.455576 7718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:42.455533 7719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:42.456470 7713 server_base.cc:1034] running on GCE node
I20260925 02:50:42.457093 7713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:42.458050 7713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:42.459268 7713 hybrid_clock.cc:648] HybridClock initialized: now 1790304642459210 us; error 53 us; skew 500 ppm
I20260925 02:50:42.459685 7713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:50:42.462071 7713 webserver.cc:460] Webserver started at http://127.2.172.66:41613/ using document root <none> and password file <none>
I20260925 02:50:42.462744 7713 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:42.462841 7713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:42.463097 7713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:42.466166 7713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "3e96dc3a375745b2900083d2ebaf8cbc"
format_stamp: "Formatted at 2026-09-25 02:50:42 on dist-test-slave-g1xh"
server_key: "3785ba868f846c72d77245eaada6390a"
server_key_iv: "9a18a1e754fc3e71b3b272da472c2462"
server_key_version: "encryptionkey@0"
I20260925 02:50:42.466939 7713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "3e96dc3a375745b2900083d2ebaf8cbc"
format_stamp: "Formatted at 2026-09-25 02:50:42 on dist-test-slave-g1xh"
server_key: "3785ba868f846c72d77245eaada6390a"
server_key_iv: "9a18a1e754fc3e71b3b272da472c2462"
server_key_version: "encryptionkey@0"
I20260925 02:50:42.472642 7713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:50:42.476629 7727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:42.478160 7713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:50:42.478317 7713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "3e96dc3a375745b2900083d2ebaf8cbc"
format_stamp: "Formatted at 2026-09-25 02:50:42 on dist-test-slave-g1xh"
server_key: "3785ba868f846c72d77245eaada6390a"
server_key_iv: "9a18a1e754fc3e71b3b272da472c2462"
server_key_version: "encryptionkey@0"
I20260925 02:50:42.478535 7713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:42.485378 7713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:42.486256 7713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:42.486501 7713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:42.487452 7713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:42.489166 7713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:42.489264 7713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:42.489351 7713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:42.489408 7713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:42.532444 7713 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:43427
I20260925 02:50:42.532527 7841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:43427 every 8 connection(s)
I20260925 02:50:42.533800 7713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:42.540514 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 7713
I20260925 02:50:42.540731 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:42.541172 2737 external_mini_cluster.cc:1428] Setting key 1daf90aca5ae4658fd586fc0878c1320
I20260925 02:50:42.542922 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:42.536739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:53807 (local address 127.2.172.126:45759)
0925 02:50:42.536943 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:42.536947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:42.538390 (+ 1443us) server_negotiation.cc:407] Connection header received
0925 02:50:42.540239 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:42.540245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:42.540332 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:42.540455 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:42.542028 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:42.542035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:42.542040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:42.542063 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:42.542108 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:42.542128 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:42.542256 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:42.542619 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 02:50:42.542676 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 02:50:42.544145 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:42.537225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:45759 (local address 127.2.172.66:53807)
0925 02:50:42.538214 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:42.538257 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:42.539927 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:42.540709 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:42.540722 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:42.540860 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:42.541607 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:50:42.541610 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:42.541649 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:50:42.541801 (+ 152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:42.542282 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:42.542306 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:42.542314 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:50:42.542682 (+ 368us) client_negotiation.cc:240] Negotiation successful
0925 02:50:42.542984 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":325,"threads_started":1}
I20260925 02:50:42.544803 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:45759
--builtin_ntp_servers=127.2.172.84:37309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 02:50:42.546167 7842 heartbeater.cc:347] Connected to a master server at 127.2.172.126:45759
I20260925 02:50:42.546552 7842 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:42.547242 7842 heartbeater.cc:511] Master 127.2.172.126:45759 requested a full tablet report, sending...
I20260925 02:50:42.549036 7520 ts_manager.cc:194] Registered new tserver with Master: 3e96dc3a375745b2900083d2ebaf8cbc (127.2.172.66:43427)
W20260925 02:50:42.879917 7845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:42.880232 7845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:42.880419 7845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:42.887959 7845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:42.888068 7845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:42.888250 7845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:50:42.897661 7845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:45759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:42.899026 7845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:42.901036 7845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:42.914408 7851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:42.914548 7850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:42.914523 7854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:42.915529 7845 server_base.cc:1034] running on GCE node
I20260925 02:50:42.916234 7845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:42.917291 7845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:42.918494 7845 hybrid_clock.cc:648] HybridClock initialized: now 1790304642918449 us; error 31 us; skew 500 ppm
I20260925 02:50:42.918824 7845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:50:42.921531 7845 webserver.cc:460] Webserver started at http://127.2.172.67:34283/ using document root <none> and password file <none>
I20260925 02:50:42.922274 7845 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:42.922365 7845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:42.922688 7845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:42.925729 7845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "217b6a2c5bbe4f138612d044f6a5473d"
format_stamp: "Formatted at 2026-09-25 02:50:42 on dist-test-slave-g1xh"
server_key: "8dca436bd429740e05aff7e38e82a159"
server_key_iv: "1de738695614986da76d575c0d55e08e"
server_key_version: "encryptionkey@0"
I20260925 02:50:42.926642 7845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "217b6a2c5bbe4f138612d044f6a5473d"
format_stamp: "Formatted at 2026-09-25 02:50:42 on dist-test-slave-g1xh"
server_key: "8dca436bd429740e05aff7e38e82a159"
server_key_iv: "1de738695614986da76d575c0d55e08e"
server_key_version: "encryptionkey@0"
I20260925 02:50:42.932132 7845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 02:50:42.936182 7860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:42.937675 7845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:50:42.937829 7845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "217b6a2c5bbe4f138612d044f6a5473d"
format_stamp: "Formatted at 2026-09-25 02:50:42 on dist-test-slave-g1xh"
server_key: "8dca436bd429740e05aff7e38e82a159"
server_key_iv: "1de738695614986da76d575c0d55e08e"
server_key_version: "encryptionkey@0"
I20260925 02:50:42.938041 7845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:42.943918 7845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:42.944747 7845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:42.945036 7845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:42.946045 7845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:42.947752 7845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:42.947866 7845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:42.947992 7845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:42.948084 7845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:42.982664 7845 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:43165
I20260925 02:50:42.982764 7972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:43165 every 8 connection(s)
I20260925 02:50:42.984302 7845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:42.987790 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 7845
I20260925 02:50:42.988039 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:42.988613 2737 external_mini_cluster.cc:1428] Setting key a7e06941fe035e242f85ddc9a4a88b73
I20260925 02:50:42.993475 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:42.987030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:48401 (local address 127.2.172.126:45759)
0925 02:50:42.987253 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:42.987259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:42.988963 (+ 1704us) server_negotiation.cc:407] Connection header received
0925 02:50:42.991051 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:42.991057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:42.991146 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:42.991268 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:42.992685 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:42.992691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:42.992696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:42.992718 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:42.992757 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:42.992768 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:42.992888 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:42.993098 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 02:50:42.993170 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:50:42.994467 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:42.987636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:45759 (local address 127.2.172.67:48401)
0925 02:50:42.988781 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:42.988827 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:42.990771 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:42.991537 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:42.991551 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:42.991706 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:42.992374 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:50:42.992376 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:42.992425 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:50:42.992511 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:42.992862 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:42.992872 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:42.992880 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:50:42.993174 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 02:50:42.993468 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":376,"threads_started":1}
I20260925 02:50:42.995945 7973 heartbeater.cc:347] Connected to a master server at 127.2.172.126:45759
I20260925 02:50:42.996284 7973 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:42.996913 7973 heartbeater.cc:511] Master 127.2.172.126:45759 requested a full tablet report, sending...
I20260925 02:50:42.998304 7520 ts_manager.cc:194] Registered new tserver with Master: 217b6a2c5bbe4f138612d044f6a5473d (127.2.172.67:43165)
I20260925 02:50:43.004477 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:50:43.011330 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.008952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.2.172.126:45759)
0925 02:50:43.009151 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:43.009157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:43.009702 (+ 545us) server_negotiation.cc:407] Connection header received
0925 02:50:43.009812 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:43.009816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:43.009877 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:43.009981 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:43.010816 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:43.010821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:43.010824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:43.010839 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:43.010879 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:43.010892 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:43.010988 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:43.011077 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 02:50:43.011122 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:50:43.015256 7520 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:43.022289 7520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:43.057701 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.038179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.2.172.67:43165)
0925 02:50:43.052808 (+ 14629us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:43.052821 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:43.054175 (+ 1354us) server_negotiation.cc:407] Connection header received
0925 02:50:43.054419 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:43.054427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:43.054616 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:43.054745 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:43.056784 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:43.056793 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:43.056807 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:43.056828 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:43.056880 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:43.056902 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:43.057075 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:43.057157 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:50:43.057271 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14395,"thread_start_us":14189,"threads_started":1}
I20260925 02:50:43.057762 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.037921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:43165 (local address 127.0.0.1:50038)
0925 02:50:43.054064 (+ 16143us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.054082 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:43.054262 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:43.055958 (+ 1696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:43.055962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:43.055984 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:43.056560 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:50:43.056564 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:43.056580 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:50:43.056622 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:43.056996 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:43.057001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:43.057008 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:50:43.057166 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 02:50:43.057220 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16038,"thread_start_us":15434,"threads_started":1}
I20260925 02:50:43.057813 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.035430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:43427 (local address 127.0.0.1:41396)
0925 02:50:43.049717 (+ 14287us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.049788 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:43.050008 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:43.054395 (+ 4387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:43.054404 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:43.054516 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:43.055654 (+ 1138us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:50:43.055657 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:43.055690 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:50:43.055804 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:43.057133 (+ 1329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:43.057142 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:43.057147 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:50:43.057282 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 02:50:43.057333 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13884,"thread_start_us":14723,"threads_started":1}
I20260925 02:50:43.058080 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.036037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.2.172.65:43983)
0925 02:50:43.053323 (+ 17286us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:43.053334 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:43.053969 (+ 635us) server_negotiation.cc:407] Connection header received
0925 02:50:43.054259 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:43.054268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:43.054475 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:43.054611 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:43.056574 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:43.056584 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:43.056599 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:43.056631 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:43.056702 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:43.056726 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:43.056879 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:43.057641 (+ 762us) server_negotiation.cc:299] Negotiation successful
0925 02:50:43.057735 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17071,"thread_start_us":16924,"threads_started":1}
I20260925 02:50:43.058478 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.035646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43983 (local address 127.0.0.1:50538)
0925 02:50:43.053827 (+ 18181us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.053854 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:43.054072 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:43.055623 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:43.055627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:43.055650 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:43.056355 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:50:43.056362 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:43.056382 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:50:43.056432 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:43.056844 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:43.056858 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:43.056865 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:50:43.057068 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 02:50:43.057163 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17865,"thread_start_us":18159,"threads_started":1}
I20260925 02:50:43.059348 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.035691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.2.172.66:43427)
0925 02:50:43.053737 (+ 18046us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:43.053746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:43.053783 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:50:43.053865 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:43.053875 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:43.054161 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:43.055985 (+ 1824us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:43.056885 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:43.056893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:43.056905 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:43.056930 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:43.056985 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:43.057007 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:43.058749 (+ 1742us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:43.058940 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 02:50:43.059038 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17894,"thread_start_us":17771,"threads_started":1}
I20260925 02:50:43.062305 7776 tablet_service.cc:1467] Processing CreateTablet for tablet 73e9fa1f4b8f4caea54c060518b10366 (DEFAULT_TABLE table=test-table [id=251f7859c9ba4c42b53c8ec5b8977751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:43.062310 7908 tablet_service.cc:1467] Processing CreateTablet for tablet 73e9fa1f4b8f4caea54c060518b10366 (DEFAULT_TABLE table=test-table [id=251f7859c9ba4c42b53c8ec5b8977751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:43.062572 7643 tablet_service.cc:1467] Processing CreateTablet for tablet 73e9fa1f4b8f4caea54c060518b10366 (DEFAULT_TABLE table=test-table [id=251f7859c9ba4c42b53c8ec5b8977751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:43.063724 7776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e9fa1f4b8f4caea54c060518b10366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:43.063746 7643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e9fa1f4b8f4caea54c060518b10366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:43.063743 7908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e9fa1f4b8f4caea54c060518b10366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:43.081634 7991 tablet_bootstrap.cc:492] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc: Bootstrap starting.
I20260925 02:50:43.085258 7992 tablet_bootstrap.cc:492] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506: Bootstrap starting.
I20260925 02:50:43.087816 7991 tablet_bootstrap.cc:654] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:43.087966 7993 tablet_bootstrap.cc:492] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d: Bootstrap starting.
I20260925 02:50:43.089227 7991 log.cc:824] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:43.091063 7993 tablet_bootstrap.cc:654] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:43.092535 7993 log.cc:824] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:43.092759 7991 tablet_bootstrap.cc:492] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc: No bootstrap required, opened a new log
I20260925 02:50:43.093042 7991 ts_tablet_manager.cc:1397] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20260925 02:50:43.094666 7709 heartbeater.cc:503] Master 127.2.172.126:45759 was elected leader, sending a full tablet report...
I20260925 02:50:43.094792 7992 tablet_bootstrap.cc:654] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:43.095978 7993 tablet_bootstrap.cc:492] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d: No bootstrap required, opened a new log
I20260925 02:50:43.096287 7993 ts_tablet_manager.cc:1397] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 02:50:43.100065 7992 log.cc:824] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:43.102114 7991 raft_consensus.cc:348] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 43427 } } peers { permanent_uuid: "d85791cfac2143e1af841ff0dabf6506" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43983 } } peers { permanent_uuid: "217b6a2c5bbe4f138612d044f6a5473d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43165 } }
I20260925 02:50:43.102520 7991 raft_consensus.cc:374] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:43.102659 7991 raft_consensus.cc:729] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e96dc3a375745b2900083d2ebaf8cbc, State: Initialized, Role: FOLLOWER
I20260925 02:50:43.103307 7991 consensus_queue.cc:260] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 43427 } } peers { permanent_uuid: "d85791cfac2143e1af841ff0dabf6506" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43983 } } peers { permanent_uuid: "217b6a2c5bbe4f138612d044f6a5473d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43165 } }
I20260925 02:50:43.103583 7992 tablet_bootstrap.cc:492] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506: No bootstrap required, opened a new log
I20260925 02:50:43.103935 7992 ts_tablet_manager.cc:1397] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.002s
I20260925 02:50:43.104434 7842 heartbeater.cc:503] Master 127.2.172.126:45759 was elected leader, sending a full tablet report...
I20260925 02:50:43.105137 7991 ts_tablet_manager.cc:1428] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260925 02:50:43.106581 7993 raft_consensus.cc:348] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 43427 } } peers { permanent_uuid: "d85791cfac2143e1af841ff0dabf6506" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43983 } } peers { permanent_uuid: "217b6a2c5bbe4f138612d044f6a5473d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43165 } }
I20260925 02:50:43.107035 7993 raft_consensus.cc:374] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:43.107187 7993 raft_consensus.cc:729] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217b6a2c5bbe4f138612d044f6a5473d, State: Initialized, Role: FOLLOWER
I20260925 02:50:43.107962 7993 consensus_queue.cc:260] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 43427 } } peers { permanent_uuid: "d85791cfac2143e1af841ff0dabf6506" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43983 } } peers { permanent_uuid: "217b6a2c5bbe4f138612d044f6a5473d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43165 } }
I20260925 02:50:43.109153 7973 heartbeater.cc:503] Master 127.2.172.126:45759 was elected leader, sending a full tablet report...
I20260925 02:50:43.109855 7993 ts_tablet_manager.cc:1428] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260925 02:50:43.114459 7992 raft_consensus.cc:348] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 43427 } } peers { permanent_uuid: "d85791cfac2143e1af841ff0dabf6506" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43983 } } peers { permanent_uuid: "217b6a2c5bbe4f138612d044f6a5473d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43165 } }
I20260925 02:50:43.114809 7992 raft_consensus.cc:374] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:43.114884 7992 raft_consensus.cc:729] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85791cfac2143e1af841ff0dabf6506, State: Initialized, Role: FOLLOWER
I20260925 02:50:43.115439 7992 consensus_queue.cc:260] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 43427 } } peers { permanent_uuid: "d85791cfac2143e1af841ff0dabf6506" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43983 } } peers { permanent_uuid: "217b6a2c5bbe4f138612d044f6a5473d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43165 } }
I20260925 02:50:43.117065 7992 ts_tablet_manager.cc:1428] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
I20260925 02:50:43.153703 7997 raft_consensus.cc:482] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:43.154008 7997 raft_consensus.cc:504] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 43427 } } peers { permanent_uuid: "d85791cfac2143e1af841ff0dabf6506" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43983 } } peers { permanent_uuid: "217b6a2c5bbe4f138612d044f6a5473d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43165 } }
I20260925 02:50:43.156044 7997 leader_election.cc:288] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d85791cfac2143e1af841ff0dabf6506 (127.2.172.65:43983), 217b6a2c5bbe4f138612d044f6a5473d (127.2.172.67:43165)
I20260925 02:50:43.160112 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.156372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:46175 (local address 127.2.172.65:43983)
0925 02:50:43.156578 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:43.156585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:43.157491 (+ 906us) server_negotiation.cc:407] Connection header received
0925 02:50:43.157721 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:43.157730 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:43.157797 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:43.157875 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:43.159379 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:43.159383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:43.159387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:43.159403 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:43.159436 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:43.159470 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:43.159627 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:43.159760 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 02:50:43.159826 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:50:43.160193 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.156573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:56537 (local address 127.2.172.67:43165)
0925 02:50:43.156758 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:43.156766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:43.157657 (+ 891us) server_negotiation.cc:407] Connection header received
0925 02:50:43.157838 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:43.157843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:43.157929 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:43.158033 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:43.159569 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:43.159578 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:43.159582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:43.159601 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:43.159641 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:43.159660 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:43.159779 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:43.159879 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 02:50:43.159934 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:50:43.160470 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.156534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:43165 (local address 127.2.172.66:56537)
0925 02:50:43.157546 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.157561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:43.157694 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:43.158090 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:43.158096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:43.158121 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:43.159376 (+ 1255us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:50:43.159378 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:43.159388 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:50:43.159412 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:43.159767 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:43.159774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:43.159785 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 02:50:43.159913 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 02:50:43.159956 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"thread_start_us":602,"threads_started":1}
I20260925 02:50:43.160753 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.156384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43983 (local address 127.2.172.66:46175)
0925 02:50:43.157361 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.157380 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:43.157541 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:43.158090 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:43.158096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:43.158121 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:43.159133 (+ 1012us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:50:43.159137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:43.159155 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:50:43.159203 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:43.159628 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:43.159633 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:43.159638 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:50:43.159805 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 02:50:43.159866 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":470,"threads_started":1}
I20260925 02:50:43.161597 7928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e9fa1f4b8f4caea54c060518b10366" candidate_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "217b6a2c5bbe4f138612d044f6a5473d" is_pre_election: true
I20260925 02:50:43.162102 7928 raft_consensus.cc:2454] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e96dc3a375745b2900083d2ebaf8cbc in term 0.
I20260925 02:50:43.161999 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e9fa1f4b8f4caea54c060518b10366" candidate_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d85791cfac2143e1af841ff0dabf6506" is_pre_election: true
I20260925 02:50:43.162622 7664 raft_consensus.cc:2454] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e96dc3a375745b2900083d2ebaf8cbc in term 0.
I20260925 02:50:43.162865 7729 leader_election.cc:302] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 217b6a2c5bbe4f138612d044f6a5473d, 3e96dc3a375745b2900083d2ebaf8cbc; no voters:
I20260925 02:50:43.163317 7997 raft_consensus.cc:2781] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:43.163482 7997 raft_consensus.cc:482] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:43.163604 7997 raft_consensus.cc:3037] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:43.165843 7997 raft_consensus.cc:504] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 43427 } } peers { permanent_uuid: "d85791cfac2143e1af841ff0dabf6506" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43983 } } peers { permanent_uuid: "217b6a2c5bbe4f138612d044f6a5473d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43165 } }
I20260925 02:50:43.166474 7997 leader_election.cc:288] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [CANDIDATE]: Term 1 election: Requested vote from peers d85791cfac2143e1af841ff0dabf6506 (127.2.172.65:43983), 217b6a2c5bbe4f138612d044f6a5473d (127.2.172.67:43165)
I20260925 02:50:43.167008 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e9fa1f4b8f4caea54c060518b10366" candidate_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d85791cfac2143e1af841ff0dabf6506"
I20260925 02:50:43.167052 7928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e9fa1f4b8f4caea54c060518b10366" candidate_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "217b6a2c5bbe4f138612d044f6a5473d"
I20260925 02:50:43.167234 7664 raft_consensus.cc:3037] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:43.167270 7928 raft_consensus.cc:3037] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:43.169546 7928 raft_consensus.cc:2454] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e96dc3a375745b2900083d2ebaf8cbc in term 1.
I20260925 02:50:43.169807 7664 raft_consensus.cc:2454] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e96dc3a375745b2900083d2ebaf8cbc in term 1.
I20260925 02:50:43.170010 7729 leader_election.cc:302] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 217b6a2c5bbe4f138612d044f6a5473d, 3e96dc3a375745b2900083d2ebaf8cbc; no voters:
I20260925 02:50:43.170390 7997 raft_consensus.cc:2781] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:43.170876 7997 raft_consensus.cc:686] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 1 LEADER]: Becoming Leader. State: Replica: 3e96dc3a375745b2900083d2ebaf8cbc, State: Running, Role: LEADER
I20260925 02:50:43.171470 7997 consensus_queue.cc:237] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 43427 } } peers { permanent_uuid: "d85791cfac2143e1af841ff0dabf6506" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43983 } } peers { permanent_uuid: "217b6a2c5bbe4f138612d044f6a5473d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43165 } }
I20260925 02:50:43.176837 7520 catalog_manager.cc:5495] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e96dc3a375745b2900083d2ebaf8cbc (127.2.172.66). New cstate: current_term: 1 leader_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e96dc3a375745b2900083d2ebaf8cbc" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 43427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d85791cfac2143e1af841ff0dabf6506" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "217b6a2c5bbe4f138612d044f6a5473d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43165 } health_report { overall_health: UNKNOWN } } }
I20260925 02:50:43.192894 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.190381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.2.172.66:43427)
0925 02:50:43.190578 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:43.190588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:43.190672 (+ 84us) server_negotiation.cc:407] Connection header received
0925 02:50:43.190960 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:43.190965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:43.191057 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:43.191180 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:43.192154 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:43.192163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:43.192173 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:43.192190 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:43.192223 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:43.192236 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:43.192425 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:43.192534 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:50:43.192600 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:50:43.201267 7928 raft_consensus.cc:1261] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d [term 1 FOLLOWER]: Refusing update from remote peer 3e96dc3a375745b2900083d2ebaf8cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:43.201300 7664 raft_consensus.cc:1261] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506 [term 1 FOLLOWER]: Refusing update from remote peer 3e96dc3a375745b2900083d2ebaf8cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:43.202323 8002 consensus_queue.cc:1035] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "217b6a2c5bbe4f138612d044f6a5473d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:43.202683 7997 consensus_queue.cc:1035] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d85791cfac2143e1af841ff0dabf6506" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:43.216447 8006 mvcc.cc:205] Tried to move back new op lower bound from 7333087818541625344 to 7333087818435215360. Current Snapshot: MvccSnapshot[applied={T|T < 7333087818541625344}]
I20260925 02:50:43.229704 7520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:43.230058 7520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:43.234129 7520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90fe37d6e7be44ae9d9e8f07241fcb7b: Sending DeleteTablet for 3 replicas of tablet 73e9fa1f4b8f4caea54c060518b10366
I20260925 02:50:43.235626 7776 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e9fa1f4b8f4caea54c060518b10366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:43 UTC) from {username='slave'} at 127.0.0.1:41396
I20260925 02:50:43.235803 7908 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e9fa1f4b8f4caea54c060518b10366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:43 UTC) from {username='slave'} at 127.0.0.1:50038
I20260925 02:50:43.236166 7643 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e9fa1f4b8f4caea54c060518b10366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:43 UTC) from {username='slave'} at 127.0.0.1:50538
I20260925 02:50:43.238101 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 7580
I20260925 02:50:43.249239 8016 tablet_replica.cc:330] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc: stopping tablet replica
I20260925 02:50:43.249935 8016 raft_consensus.cc:2229] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:50:43.250737 8016 raft_consensus.cc:2258] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:43.253206 8016 ts_tablet_manager.cc:1905] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:43.253845 8017 tablet_replica.cc:330] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506: stopping tablet replica
I20260925 02:50:43.254351 8017 raft_consensus.cc:2229] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:43.259011 8018 tablet_replica.cc:330] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d: stopping tablet replica
I20260925 02:50:43.259347 8017 raft_consensus.cc:2258] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:43.260349 8018 raft_consensus.cc:2229] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:43.260699 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.240067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.2.172.65:43983)
0925 02:50:43.240352 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:43.240362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:43.256717 (+ 16355us) server_negotiation.cc:407] Connection header received
0925 02:50:43.257389 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:43.257401 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:43.257552 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:43.257657 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:43.259709 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:43.259715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:43.259720 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:43.259744 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:43.259787 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:43.259804 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:43.259917 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:43.260185 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 02:50:43.260384 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:50:43.260913 8018 raft_consensus.cc:2258] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:43.262042 7704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:43.262938 8016 ts_tablet_manager.cc:1918] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:43.263298 8016 log.cc:1196] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/73e9fa1f4b8f4caea54c060518b10366
I20260925 02:50:43.264008 8016 ts_tablet_manager.cc:1939] T 73e9fa1f4b8f4caea54c060518b10366 P 3e96dc3a375745b2900083d2ebaf8cbc: Deleting consensus metadata
I20260925 02:50:43.264688 8018 ts_tablet_manager.cc:1905] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:43.266211 7507 catalog_manager.cc:4841] TS 3e96dc3a375745b2900083d2ebaf8cbc (127.2.172.66:43427): tablet 73e9fa1f4b8f4caea54c060518b10366 (table test-table [id=251f7859c9ba4c42b53c8ec5b8977751]) successfully deleted
I20260925 02:50:43.270112 8018 ts_tablet_manager.cc:1918] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:43.270320 8018 log.cc:1196] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/73e9fa1f4b8f4caea54c060518b10366
I20260925 02:50:43.271072 8018 ts_tablet_manager.cc:1939] T 73e9fa1f4b8f4caea54c060518b10366 P 217b6a2c5bbe4f138612d044f6a5473d: Deleting consensus metadata
I20260925 02:50:43.273480 7505 catalog_manager.cc:4841] TS 217b6a2c5bbe4f138612d044f6a5473d (127.2.172.67:43165): tablet 73e9fa1f4b8f4caea54c060518b10366 (table test-table [id=251f7859c9ba4c42b53c8ec5b8977751]) successfully deleted
I20260925 02:50:43.263350 8017 ts_tablet_manager.cc:1905] T 73e9fa1f4b8f4caea54c060518b10366 P d85791cfac2143e1af841ff0dabf6506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:43.360006 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 7580
I20260925 02:50:43.379873 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 7713
W20260925 02:50:43.379966 7504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.172.65:43983 (error 108)
I20260925 02:50:43.382433 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.381799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43983 (local address 127.0.0.1:50556)
0925 02:50:43.382004 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.382169 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260925 02:50:43.382740 7504 catalog_manager.cc:4584] TS d85791cfac2143e1af841ff0dabf6506 (127.2.172.65:43983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73e9fa1f4b8f4caea54c060518b10366: Network error: Client connection negotiation failed: client connection to 127.2.172.65:43983: connect: Connection refused (error 111)
I20260925 02:50:43.384232 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.381211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.2.172.66:43427)
0925 02:50:43.381825 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:43.381835 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:43.381859 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:50:43.381955 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:43.381960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:43.382041 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:43.382169 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:43.383419 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:43.383426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:43.383431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:43.383475 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:43.383518 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:43.383535 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:43.383698 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:43.383816 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 02:50:43.383867 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470}
I20260925 02:50:43.385066 7837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:43.436668 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.436077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43983 (local address 127.0.0.1:50568)
0925 02:50:43.436342 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.436473 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 02:50:43.437793 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.437405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43983 (local address 127.0.0.1:50574)
0925 02:50:43.437560 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.437660 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 02:50:43.498546 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.498027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43983 (local address 127.0.0.1:50576)
0925 02:50:43.498255 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.498390 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 02:50:43.500530 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 7713
I20260925 02:50:43.500960 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.500509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43983 (local address 127.0.0.1:50592)
0925 02:50:43.500712 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.500800 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 02:50:43.520960 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 7845
I20260925 02:50:43.524293 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.521858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.2.172.67:43165)
0925 02:50:43.522044 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:43.522051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:43.522152 (+ 101us) server_negotiation.cc:407] Connection header received
0925 02:50:43.522408 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:43.522414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:43.522535 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:43.522657 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:43.523614 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:43.523620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:43.523624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:43.523643 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:43.523701 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:43.523718 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:43.523851 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:43.523929 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 02:50:43.523991 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:50:43.524958 7968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:43.581833 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.581307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43983 (local address 127.0.0.1:50606)
0925 02:50:43.581582 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.581683 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260925 02:50:43.582932 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.582499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:43983 (local address 127.0.0.1:50622)
0925 02:50:43.582736 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.582804 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 02:50:43.621825 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 7845
I20260925 02:50:43.639905 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 7489
I20260925 02:50:43.641289 7550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:43.748044 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 7489
2026-09-25T02:50:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2745 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 02:50:43.791896 2737 mini_oidc.cc:152] Starting JWKS server
I20260925 02:50:43.794975 2737 webserver.cc:460] Webserver started at https://127.0.0.1:39895/ using document root <none> and password file <none>
I20260925 02:50:43.795356 2737 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:50:43.796152 2737 webserver.cc:460] Webserver started at http://127.0.0.1:36659/ using document root <none> and password file <none>
2026-09-25T02:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:43Z Disabled control of system clock
I20260925 02:50:43.826741 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:39319
--webserver_interface=127.2.172.126
--webserver_port=0
--builtin_ntp_servers=127.2.172.84:40973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:39319
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39895/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/testchainca.cert with env {}
W20260925 02:50:44.152205 8030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:44.152550 8030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:44.152714 8030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:44.160348 8030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:44.160509 8030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:44.160599 8030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:44.160705 8030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:44.160794 8030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:50:44.160882 8030 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:50:44.160964 8030 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39895/jwks.json
I20260925 02:50:44.171161 8030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:39319
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:39319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39895/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:44.172592 8030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:44.173926 8030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:44.185917 8038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:44.186195 8036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:44.186404 8035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:44.186903 8030 server_base.cc:1034] running on GCE node
I20260925 02:50:44.187863 8030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:44.189397 8030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:44.190634 8030 hybrid_clock.cc:648] HybridClock initialized: now 1790304644190574 us; error 42 us; skew 500 ppm
I20260925 02:50:44.190990 8030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:50:44.193338 8030 webserver.cc:460] Webserver started at http://127.2.172.126:46873/ using document root <none> and password file <none>
I20260925 02:50:44.194024 8030 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:44.194128 8030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:44.194391 8030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:44.197440 8030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "1bd67598fe80482a8a20c7806c344f2c"
format_stamp: "Formatted at 2026-09-25 02:50:44 on dist-test-slave-g1xh"
server_key: "061380ce9c3f52b2f5798d6a4fa4d758"
server_key_iv: "374fac3b4c277da174c7e1893709a38b"
server_key_version: "encryptionkey@0"
I20260925 02:50:44.198185 8030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "1bd67598fe80482a8a20c7806c344f2c"
format_stamp: "Formatted at 2026-09-25 02:50:44 on dist-test-slave-g1xh"
server_key: "061380ce9c3f52b2f5798d6a4fa4d758"
server_key_iv: "374fac3b4c277da174c7e1893709a38b"
server_key_version: "encryptionkey@0"
I20260925 02:50:44.203730 8030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260925 02:50:44.207252 8044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:44.208702 8030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:44.208886 8030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "1bd67598fe80482a8a20c7806c344f2c"
format_stamp: "Formatted at 2026-09-25 02:50:44 on dist-test-slave-g1xh"
server_key: "061380ce9c3f52b2f5798d6a4fa4d758"
server_key_iv: "374fac3b4c277da174c7e1893709a38b"
server_key_version: "encryptionkey@0"
I20260925 02:50:44.209125 8030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:44.231384 8030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:44.232302 8030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:44.232645 8030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:44.251132 8030 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:39319
I20260925 02:50:44.251155 8095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:39319 every 8 connection(s)
I20260925 02:50:44.252576 8030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:44.256520 8096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:44.257253 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 8030
I20260925 02:50:44.257471 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:44.258287 2737 external_mini_cluster.cc:1428] Setting key 2c39aae4b6157898df53a740658efd72
I20260925 02:50:44.270789 8096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c: Bootstrap starting.
I20260925 02:50:44.274766 8096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:44.275620 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:44.260536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.2.172.126:39319)
0925 02:50:44.261964 (+ 1428us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:44.261989 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:44.262621 (+ 632us) server_negotiation.cc:407] Connection header received
0925 02:50:44.265099 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:44.265130 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:44.265728 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:44.266163 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:44.266745 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:44.268813 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:44.270164 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:44.271194 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:44.271408 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:44.272648 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:44.272662 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:44.272680 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:44.273405 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:44.273524 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:44.273551 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:44.273676 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:44.273882 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 02:50:44.274198 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":428,"threads_started":1}
I20260925 02:50:44.276177 8096 log.cc:824] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:44.279304 8096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c: No bootstrap required, opened a new log
I20260925 02:50:44.279702 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:50:44.287077 8096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd67598fe80482a8a20c7806c344f2c" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39319 } }
I20260925 02:50:44.287509 8096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:44.287637 8096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd67598fe80482a8a20c7806c344f2c, State: Initialized, Role: FOLLOWER
I20260925 02:50:44.288251 8096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd67598fe80482a8a20c7806c344f2c" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39319 } }
I20260925 02:50:44.288578 8096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:44.288735 8096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:44.288945 8096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:44.291185 8096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd67598fe80482a8a20c7806c344f2c" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39319 } }
I20260925 02:50:44.291704 8096 leader_election.cc:302] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bd67598fe80482a8a20c7806c344f2c; no voters:
I20260925 02:50:44.292236 8096 leader_election.cc:288] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:44.292392 8101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:44.293177 8101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [term 1 LEADER]: Becoming Leader. State: Replica: 1bd67598fe80482a8a20c7806c344f2c, State: Running, Role: LEADER
I20260925 02:50:44.293807 8101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd67598fe80482a8a20c7806c344f2c" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39319 } }
I20260925 02:50:44.294687 8096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:44.297044 8102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bd67598fe80482a8a20c7806c344f2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd67598fe80482a8a20c7806c344f2c" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39319 } } }
I20260925 02:50:44.297019 8103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bd67598fe80482a8a20c7806c344f2c. Latest consensus state: current_term: 1 leader_uuid: "1bd67598fe80482a8a20c7806c344f2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd67598fe80482a8a20c7806c344f2c" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39319 } } }
I20260925 02:50:44.297475 8102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:44.297614 8103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:44.299890 8108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:44.304674 8108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:44.314805 8108 catalog_manager.cc:1347] Generated new cluster ID: fca53e0374b84cb19386eb6f5dda1027
I20260925 02:50:44.314937 8108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:44.337617 8108 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:44.338996 8108 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:44.345541 8108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bd67598fe80482a8a20c7806c344f2c: Generated new TSK 0
I20260925 02:50:44.346633 8108 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:50:44.398658 8126 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 02:50:44.398895 8126 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:50:44.400183 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:44.376464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.2.172.126:39319)
0925 02:50:44.376679 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:44.376686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:44.377433 (+ 747us) server_negotiation.cc:407] Connection header received
0925 02:50:44.377728 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:44.377734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:44.377829 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:44.378008 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:50:44.378576 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:44.379758 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:44.381478 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:44.382187 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:44.382319 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:44.382870 (+ 551us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:50:44.399392 (+ 16522us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:50:44.399830 (+ 438us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52702: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 02:50:44.400331 8099 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52702: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 02:50:44.402343 8125 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:50:44.376458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39319 (local address 127.0.0.1:52702)
0925 02:50:44.377314 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:44.377332 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:44.377497 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:44.378048 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:44.378052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:44.378063 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:50:44.378418 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:44.378428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:44.379948 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:44.379953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:44.381301 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:44.381313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:44.382357 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:44.382363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:44.382509 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:44.382578 (+ 69us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:50:44.399835 (+ 17257us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:50:44.400417 (+ 582us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.172.126:39319: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":470,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.172.126:39319: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 02:50:44.404167 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 8030
I20260925 02:50:44.405689 8091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:44.502231 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 8030
2026-09-25T02:50:44Z chronyd exiting
I20260925 02:50:44.553028 2737 test_util.cc:175] -----------------------------------------------
I20260925 02:50:44.553154 2737 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304596269475-2737-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (766 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 02:50:44.556887 2737 mini_oidc.cc:152] Starting JWKS server
I20260925 02:50:44.558781 2737 webserver.cc:460] Webserver started at https://127.0.0.1:39173/ using document root <none> and password file <none>
I20260925 02:50:44.559078 2737 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:50:44.559736 2737 webserver.cc:460] Webserver started at http://127.0.0.1:38081/ using document root <none> and password file <none>
2026-09-25T02:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:44Z Disabled control of system clock
I20260925 02:50:44.590687 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:45849
--webserver_interface=127.2.172.126
--webserver_port=0
--builtin_ntp_servers=127.2.172.84:37077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:45849
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39173/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/testchainca.cert with env {}
W20260925 02:50:44.920431 8133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:44.920785 8133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:44.920956 8133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:44.928467 8133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:44.928617 8133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:44.928706 8133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:44.928817 8133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:44.928910 8133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:50:44.929003 8133 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:50:44.929082 8133 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39173/jwks.json
I20260925 02:50:44.939231 8133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:45849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:45849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39173/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:44.940632 8133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:44.942376 8133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:44.954632 8138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:44.954555 8141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:44.954598 8139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:44.955710 8133 server_base.cc:1034] running on GCE node
I20260925 02:50:44.956545 8133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:44.958010 8133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:44.959260 8133 hybrid_clock.cc:648] HybridClock initialized: now 1790304644959204 us; error 33 us; skew 500 ppm
I20260925 02:50:44.959699 8133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:50:44.962193 8133 webserver.cc:460] Webserver started at http://127.2.172.126:37755/ using document root <none> and password file <none>
I20260925 02:50:44.962920 8133 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:44.963063 8133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:44.963440 8133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:44.966353 8133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "1401a5cc140c4617a6b2081f2bf42e9d"
format_stamp: "Formatted at 2026-09-25 02:50:44 on dist-test-slave-g1xh"
server_key: "aa5f6b342439fe2663281c2f458843f2"
server_key_iv: "fce290ad954e429645e1e5256b617265"
server_key_version: "encryptionkey@0"
I20260925 02:50:44.967150 8133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "1401a5cc140c4617a6b2081f2bf42e9d"
format_stamp: "Formatted at 2026-09-25 02:50:44 on dist-test-slave-g1xh"
server_key: "aa5f6b342439fe2663281c2f458843f2"
server_key_iv: "fce290ad954e429645e1e5256b617265"
server_key_version: "encryptionkey@0"
I20260925 02:50:44.972342 8133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 02:50:44.975858 8147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:44.977310 8133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:50:44.977520 8133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "1401a5cc140c4617a6b2081f2bf42e9d"
format_stamp: "Formatted at 2026-09-25 02:50:44 on dist-test-slave-g1xh"
server_key: "aa5f6b342439fe2663281c2f458843f2"
server_key_iv: "fce290ad954e429645e1e5256b617265"
server_key_version: "encryptionkey@0"
I20260925 02:50:44.977752 8133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:45.011791 8133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:45.012676 8133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:45.013000 8133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:45.032290 8133 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:45849
I20260925 02:50:45.032392 8200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:45849 every 8 connection(s)
I20260925 02:50:45.033686 8133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:45.037312 8201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:45.043090 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 8133
I20260925 02:50:45.043321 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:45.044149 2737 external_mini_cluster.cc:1428] Setting key 8075411e0e13d40c490236056fa269d8
I20260925 02:50:45.048969 8201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d: Bootstrap starting.
I20260925 02:50:45.052116 8201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:45.053128 8201 log.cc:824] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:45.056581 8201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d: No bootstrap required, opened a new log
I20260925 02:50:45.058197 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:45.046317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.2.172.126:45849)
0925 02:50:45.047308 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:45.047319 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:45.047359 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:50:45.049128 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:45.049154 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:45.049638 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:45.050032 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:45.050607 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:45.051938 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:45.053318 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:45.054491 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:45.054685 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:45.055898 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:45.055906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:45.055920 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:45.056458 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:45.056534 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:45.056555 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:45.056645 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:45.056911 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 02:50:45.057273 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":317,"threads_started":1}
I20260925 02:50:45.062285 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:50:45.064862 8201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1401a5cc140c4617a6b2081f2bf42e9d" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45849 } }
I20260925 02:50:45.065243 8201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:45.065380 8201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1401a5cc140c4617a6b2081f2bf42e9d, State: Initialized, Role: FOLLOWER
I20260925 02:50:45.065986 8201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1401a5cc140c4617a6b2081f2bf42e9d" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45849 } }
I20260925 02:50:45.066191 8201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:45.066291 8201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:45.066411 8201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:45.068658 8201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1401a5cc140c4617a6b2081f2bf42e9d" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45849 } }
I20260925 02:50:45.069165 8201 leader_election.cc:302] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1401a5cc140c4617a6b2081f2bf42e9d; no voters:
I20260925 02:50:45.069681 8201 leader_election.cc:288] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:45.069839 8206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:45.070953 8206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [term 1 LEADER]: Becoming Leader. State: Replica: 1401a5cc140c4617a6b2081f2bf42e9d, State: Running, Role: LEADER
I20260925 02:50:45.071524 8206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1401a5cc140c4617a6b2081f2bf42e9d" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45849 } }
I20260925 02:50:45.072016 8201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:45.074759 8207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1401a5cc140c4617a6b2081f2bf42e9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1401a5cc140c4617a6b2081f2bf42e9d" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45849 } } }
I20260925 02:50:45.074712 8208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1401a5cc140c4617a6b2081f2bf42e9d. Latest consensus state: current_term: 1 leader_uuid: "1401a5cc140c4617a6b2081f2bf42e9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1401a5cc140c4617a6b2081f2bf42e9d" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45849 } } }
I20260925 02:50:45.075197 8207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:45.075232 8208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:45.081619 8213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:45.087373 8213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:45.097615 8213 catalog_manager.cc:1347] Generated new cluster ID: c0aa23e13f9541009787e83be1b80dad
I20260925 02:50:45.097775 8213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:45.112769 8213 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:45.114102 8213 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:45.123600 8213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1401a5cc140c4617a6b2081f2bf42e9d: Generated new TSK 0
I20260925 02:50:45.124527 8213 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:50:45.159962 8231 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 02:50:45.160095 8231 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:50:45.161489 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:45.140625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34644 (local address 127.2.172.126:45849)
0925 02:50:45.140836 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:45.140844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:45.141216 (+ 372us) server_negotiation.cc:407] Connection header received
0925 02:50:45.141410 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:45.141415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:45.141520 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:45.141700 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:50:45.142123 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:45.143290 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:45.144734 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:45.145447 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:45.145570 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:45.146064 (+ 494us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:50:45.160744 (+ 14680us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:50:45.161154 (+ 410us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 02:50:45.161613 8204 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 02:50:45.161703 8230 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:50:45.140441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:45849 (local address 127.0.0.1:34644)
0925 02:50:45.141102 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:45.141123 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:45.141241 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:45.141712 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:45.141715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:45.141724 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:50:45.141965 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:45.141972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:45.143450 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:45.143453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:45.144592 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:45.144604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:45.145621 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:45.145625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:45.145753 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:45.145784 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:50:45.160988 (+ 15204us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:50:45.161196 (+ 208us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.172.126:45849: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":378,"threads_started":1}
I20260925 02:50:45.163255 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 8133
I20260925 02:50:45.164525 8196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:45.257372 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 8133
2026-09-25T02:50:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (756 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 02:50:45.313975 2737 mini_oidc.cc:152] Starting JWKS server
I20260925 02:50:45.315858 2737 webserver.cc:460] Webserver started at https://127.0.0.1:37933/ using document root <none> and password file <none>
I20260925 02:50:45.316051 2737 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:50:45.316606 2737 webserver.cc:460] Webserver started at http://127.0.0.1:41291/ using document root <none> and password file <none>
2026-09-25T02:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:45Z Disabled control of system clock
I20260925 02:50:45.345295 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:43861
--webserver_interface=127.2.172.126
--webserver_port=0
--builtin_ntp_servers=127.2.172.84:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:43861
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37933/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:50:45.678280 8238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:45.678661 8238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:45.678823 8238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:45.686779 8238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:45.686933 8238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:45.687000 8238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:45.687079 8238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:45.687166 8238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:50:45.687254 8238 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:50:45.687331 8238 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37933/jwks.json
I20260925 02:50:45.696895 8238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:43861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:43861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37933/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:45.698299 8238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:45.700196 8238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:45.713116 8243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:45.713196 8244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:45.713217 8246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:45.714255 8238 server_base.cc:1034] running on GCE node
I20260925 02:50:45.715137 8238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:45.716794 8238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:45.718003 8238 hybrid_clock.cc:648] HybridClock initialized: now 1790304645717946 us; error 45 us; skew 500 ppm
I20260925 02:50:45.718374 8238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:50:45.729530 8238 webserver.cc:460] Webserver started at http://127.2.172.126:44723/ using document root <none> and password file <none>
I20260925 02:50:45.730316 8238 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:45.730412 8238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:45.730760 8238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:45.733839 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "9fcd9c659d7847d5a8054b4b91692460"
format_stamp: "Formatted at 2026-09-25 02:50:45 on dist-test-slave-g1xh"
server_key: "06d07bb5ff320e36966506267e74d368"
server_key_iv: "600525942025fb6a563d71e15a40bf8e"
server_key_version: "encryptionkey@0"
I20260925 02:50:45.734628 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "9fcd9c659d7847d5a8054b4b91692460"
format_stamp: "Formatted at 2026-09-25 02:50:45 on dist-test-slave-g1xh"
server_key: "06d07bb5ff320e36966506267e74d368"
server_key_iv: "600525942025fb6a563d71e15a40bf8e"
server_key_version: "encryptionkey@0"
I20260925 02:50:45.740451 8238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:50:45.744206 8252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:45.745798 8238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:50:45.746028 8238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "9fcd9c659d7847d5a8054b4b91692460"
format_stamp: "Formatted at 2026-09-25 02:50:45 on dist-test-slave-g1xh"
server_key: "06d07bb5ff320e36966506267e74d368"
server_key_iv: "600525942025fb6a563d71e15a40bf8e"
server_key_version: "encryptionkey@0"
I20260925 02:50:45.746273 8238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:45.767119 8238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:45.768138 8238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:45.768477 8238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:45.788076 8238 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:43861
I20260925 02:50:45.788102 8303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:43861 every 8 connection(s)
I20260925 02:50:45.789443 8238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:45.793251 8304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:45.796025 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 8238
I20260925 02:50:45.796219 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:45.797088 2737 external_mini_cluster.cc:1428] Setting key 2cfa519fd518241cbc4f2c0c545ef942
I20260925 02:50:45.805781 8304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460: Bootstrap starting.
I20260925 02:50:45.809062 8304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:45.810262 8304 log.cc:824] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:45.811343 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:45.799289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.2.172.126:43861)
0925 02:50:45.800447 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:45.800459 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:45.800505 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:50:45.802196 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:45.802222 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:45.802720 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:45.803171 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:45.803822 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:45.805109 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:45.806514 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:45.807218 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:45.807437 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:45.808630 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:45.808639 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:45.808653 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:45.809220 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:45.809348 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:45.809371 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:45.809516 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:45.809731 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 02:50:45.810056 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":430,"threads_started":1}
I20260925 02:50:45.813759 8304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460: No bootstrap required, opened a new log
I20260925 02:50:45.815456 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:50:45.821816 8304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd9c659d7847d5a8054b4b91692460" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43861 } }
I20260925 02:50:45.822214 8304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:45.822336 8304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fcd9c659d7847d5a8054b4b91692460, State: Initialized, Role: FOLLOWER
I20260925 02:50:45.822907 8304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd9c659d7847d5a8054b4b91692460" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43861 } }
I20260925 02:50:45.823155 8304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:45.823311 8304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:45.823485 8304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:45.825907 8304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd9c659d7847d5a8054b4b91692460" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43861 } }
I20260925 02:50:45.826422 8304 leader_election.cc:302] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fcd9c659d7847d5a8054b4b91692460; no voters:
I20260925 02:50:45.827018 8304 leader_election.cc:288] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:45.827171 8309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:45.827960 8309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [term 1 LEADER]: Becoming Leader. State: Replica: 9fcd9c659d7847d5a8054b4b91692460, State: Running, Role: LEADER
I20260925 02:50:45.828519 8309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd9c659d7847d5a8054b4b91692460" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43861 } }
I20260925 02:50:45.829224 8304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:45.831449 8311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fcd9c659d7847d5a8054b4b91692460. Latest consensus state: current_term: 1 leader_uuid: "9fcd9c659d7847d5a8054b4b91692460" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd9c659d7847d5a8054b4b91692460" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43861 } } }
I20260925 02:50:45.831882 8311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:45.833158 8310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fcd9c659d7847d5a8054b4b91692460" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd9c659d7847d5a8054b4b91692460" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43861 } } }
I20260925 02:50:45.833590 8310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:45.834154 8318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:45.839157 8318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:45.847707 8318 catalog_manager.cc:1347] Generated new cluster ID: f7bea8ccf4924894a7ff684df4f6e9e2
I20260925 02:50:45.847839 8318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:45.870303 8318 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:45.871708 8318 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:45.886088 8318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fcd9c659d7847d5a8054b4b91692460: Generated new TSK 0
I20260925 02:50:45.887007 8318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:50:45.913113 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:45.904574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.2.172.126:43861)
0925 02:50:45.904785 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:45.904792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:45.905161 (+ 369us) server_negotiation.cc:407] Connection header received
0925 02:50:45.905406 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:45.905414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:45.905510 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:45.905692 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:50:45.906106 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:45.907284 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:45.908722 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:45.909423 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:45.909551 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:45.910033 (+ 482us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:50:45.912434 (+ 2401us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37933/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:50:45.912796 (+ 362us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47742: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37933/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 02:50:45.913246 8307 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47742: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37933/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 02:50:45.914824 8333 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:50:45.904393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43861 (local address 127.0.0.1:47742)
0925 02:50:45.905054 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:45.905071 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:45.905200 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:45.905707 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:45.905711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:45.905721 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:50:45.905964 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:45.905972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:45.907453 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:45.907457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:45.908584 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:45.908596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:45.909603 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:45.909607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:45.909743 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:45.909776 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:50:45.912656 (+ 2880us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37933/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:50:45.914396 (+ 1740us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.172.126:43861: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37933/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":1415,"thread_start_us":375,"threads_started":1}
I20260925 02:50:45.916178 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 8238
I20260925 02:50:45.917511 8299 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 02:50:45.924959 8334 webserver.cc:533] Webserver: sslize failed:
E20260925 02:50:45.925081 8334 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:50:46.010478 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 8238
2026-09-25T02:50:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (752 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T02:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:46Z Disabled control of system clock
I20260925 02:50:46.096586 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:44679
--webserver_interface=127.2.172.126
--webserver_port=0
--builtin_ntp_servers=127.2.172.84:42385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:44679
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/testchainca.cert with env {}
W20260925 02:50:46.430931 8339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:46.431246 8339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:46.431346 8339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:46.438701 8339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:46.438814 8339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:46.438853 8339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:46.438911 8339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:46.438949 8339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:50:46.439011 8339 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:50:46.439050 8339 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 02:50:46.449183 8339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:42385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:44679
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:44679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:46.450528 8339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:46.451803 8339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:46.463948 8344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:46.464093 8347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:46.464066 8345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:46.464684 8339 server_base.cc:1034] running on GCE node
I20260925 02:50:46.465588 8339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:46.467078 8339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:46.468288 8339 hybrid_clock.cc:648] HybridClock initialized: now 1790304646468223 us; error 51 us; skew 500 ppm
I20260925 02:50:46.468667 8339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:50:46.471061 8339 webserver.cc:460] Webserver started at http://127.2.172.126:35337/ using document root <none> and password file <none>
I20260925 02:50:46.471884 8339 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:46.472012 8339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:46.472350 8339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:46.475337 8339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "bccffc4cbc7f490a9b6e22f4a977657a"
format_stamp: "Formatted at 2026-09-25 02:50:46 on dist-test-slave-g1xh"
server_key: "db22d3c794ba0629014d4d35eaff1419"
server_key_iv: "ffa3c386867dfb8c805827ae82f013e7"
server_key_version: "encryptionkey@0"
I20260925 02:50:46.476279 8339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "bccffc4cbc7f490a9b6e22f4a977657a"
format_stamp: "Formatted at 2026-09-25 02:50:46 on dist-test-slave-g1xh"
server_key: "db22d3c794ba0629014d4d35eaff1419"
server_key_iv: "ffa3c386867dfb8c805827ae82f013e7"
server_key_version: "encryptionkey@0"
I20260925 02:50:46.482013 8339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:50:46.486018 8353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:46.487793 8339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:46.488015 8339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "bccffc4cbc7f490a9b6e22f4a977657a"
format_stamp: "Formatted at 2026-09-25 02:50:46 on dist-test-slave-g1xh"
server_key: "db22d3c794ba0629014d4d35eaff1419"
server_key_iv: "ffa3c386867dfb8c805827ae82f013e7"
server_key_version: "encryptionkey@0"
I20260925 02:50:46.488274 8339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:46.513741 8339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:46.514753 8339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:46.515082 8339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:46.535454 8339 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:44679
I20260925 02:50:46.535461 8407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:44679 every 8 connection(s)
I20260925 02:50:46.536988 8339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:46.538312 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 8339
I20260925 02:50:46.538551 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:46.539218 2737 external_mini_cluster.cc:1428] Setting key f108f9edbe902c032b67671fc0d53e33
I20260925 02:50:46.541714 8408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:46.553365 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:46.541181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.2.172.126:44679)
0925 02:50:46.542300 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:46.542311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:46.542356 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:50:46.544097 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:46.544121 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:46.544601 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:46.545058 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:46.545641 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:46.547041 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:46.548546 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:46.549239 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:46.549445 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:46.550802 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:46.550810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:46.550861 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:50:46.551467 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:46.551603 (+ 136us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:50:46.551642 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:46.551777 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:46.552019 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 02:50:46.552362 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":416,"threads_started":1}
I20260925 02:50:46.553473 8408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a: Bootstrap starting.
I20260925 02:50:46.556918 8408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:46.557442 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:50:46.558091 8408 log.cc:824] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:46.561328 8408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a: No bootstrap required, opened a new log
I20260925 02:50:46.569286 8408 raft_consensus.cc:348] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccffc4cbc7f490a9b6e22f4a977657a" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44679 } }
I20260925 02:50:46.569778 8408 raft_consensus.cc:374] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:46.569907 8408 raft_consensus.cc:729] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccffc4cbc7f490a9b6e22f4a977657a, State: Initialized, Role: FOLLOWER
I20260925 02:50:46.570574 8408 consensus_queue.cc:260] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccffc4cbc7f490a9b6e22f4a977657a" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44679 } }
I20260925 02:50:46.570833 8408 raft_consensus.cc:388] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:46.571013 8408 raft_consensus.cc:482] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:46.571199 8408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:46.573518 8408 raft_consensus.cc:504] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccffc4cbc7f490a9b6e22f4a977657a" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44679 } }
I20260925 02:50:46.574062 8408 leader_election.cc:302] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bccffc4cbc7f490a9b6e22f4a977657a; no voters:
I20260925 02:50:46.574662 8408 leader_election.cc:288] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:46.574867 8413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:46.575919 8413 raft_consensus.cc:686] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [term 1 LEADER]: Becoming Leader. State: Replica: bccffc4cbc7f490a9b6e22f4a977657a, State: Running, Role: LEADER
I20260925 02:50:46.576751 8413 consensus_queue.cc:237] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccffc4cbc7f490a9b6e22f4a977657a" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44679 } }
I20260925 02:50:46.577082 8408 sys_catalog.cc:564] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:46.580631 8415 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bccffc4cbc7f490a9b6e22f4a977657a. Latest consensus state: current_term: 1 leader_uuid: "bccffc4cbc7f490a9b6e22f4a977657a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccffc4cbc7f490a9b6e22f4a977657a" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44679 } } }
I20260925 02:50:46.580722 8414 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bccffc4cbc7f490a9b6e22f4a977657a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccffc4cbc7f490a9b6e22f4a977657a" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44679 } } }
I20260925 02:50:46.581135 8415 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:46.581135 8414 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:46.582840 8423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:46.588155 8423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:46.598439 8423 catalog_manager.cc:1347] Generated new cluster ID: 3fe5e61a8534466595a2313888d0c631
I20260925 02:50:46.598575 8423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:46.615346 8423 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:46.616823 8423 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:46.627688 8423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bccffc4cbc7f490a9b6e22f4a977657a: Generated new TSK 0
I20260925 02:50:46.628655 8423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:50:46.731755 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:46.647743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.2.172.126:44679)
0925 02:50:46.648007 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:46.648015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:46.648436 (+ 421us) server_negotiation.cc:407] Connection header received
0925 02:50:46.648672 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:46.648676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:46.648749 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:46.648888 (+ 139us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:50:46.649476 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:46.650244 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:46.652763 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:46.653229 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:46.653332 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:46.654005 (+ 673us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:50:46.731072 (+ 77067us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:50:46.731432 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53026: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 02:50:46.731925 8411 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53026: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 02:50:46.732632 8437 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:50:46.647613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:44679 (local address 127.0.0.1:53026)
0925 02:50:46.648286 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:46.648308 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:46.648480 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:46.648940 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:46.648946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:46.648962 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:50:46.649311 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:46.649324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:46.650403 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:46.650410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:46.652586 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:46.652604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:46.653439 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:46.653447 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:46.653674 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:46.653722 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:50:46.731435 (+ 77713us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:50:46.731817 (+ 382us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.172.126:44679: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":346,"threads_started":1}
I20260925 02:50:46.733868 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 8339
I20260925 02:50:46.735129 8403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:46.835813 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 8339
2026-09-25T02:50:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (804 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (422 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (418 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:47 dist-test-slave-g1xh krb5kdc[8452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:47 dist-test-slave-g1xh krb5kdc[8452](info): set up 2 sockets
Sep 25 02:50:47 dist-test-slave-g1xh krb5kdc[8452](info): commencing operation
krb5kdc: starting...
W20260925 02:50:48.779284 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:50:48 dist-test-slave-g1xh krb5kdc[8452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:50:49.011240 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:45023
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:45023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:49.337707 8465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:49.337996 8465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:49.338141 8465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:49.345422 8465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:49.345530 8465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:49.345647 8465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:49.345757 8465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:49.345836 8465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:49.355862 8465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:45023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:45023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:49.357323 8465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:49.358678 8465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:49.371063 8470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:49.371039 8471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:49.370997 8473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:49.371860 8465 server_base.cc:1034] running on GCE node
I20260925 02:50:49.372637 8465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:49.373993 8465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:49.375217 8465 hybrid_clock.cc:648] HybridClock initialized: now 1790304649375132 us; error 78 us; skew 500 ppm
I20260925 02:50:49.375660 8465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:49 dist-test-slave-g1xh krb5kdc[8452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304649, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:49.378765 8465 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:50:49.380334 8465 webserver.cc:460] Webserver started at http://127.2.172.126:41073/ using document root <none> and password file <none>
I20260925 02:50:49.381049 8465 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:49.381198 8465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:49.381526 8465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:49.384635 8465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "c1949806c8084d2fb41582e9dfdc7d8f"
format_stamp: "Formatted at 2026-09-25 02:50:49 on dist-test-slave-g1xh"
server_key: "0ff3bd0b2e6c2106774d5fb5c8c0aa9c"
server_key_iv: "aa9d6c29769a0ac34a629e3c9d0d0de5"
server_key_version: "encryptionkey@0"
I20260925 02:50:49.385440 8465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "c1949806c8084d2fb41582e9dfdc7d8f"
format_stamp: "Formatted at 2026-09-25 02:50:49 on dist-test-slave-g1xh"
server_key: "0ff3bd0b2e6c2106774d5fb5c8c0aa9c"
server_key_iv: "aa9d6c29769a0ac34a629e3c9d0d0de5"
server_key_version: "encryptionkey@0"
I20260925 02:50:49.390699 8465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:50:49.394429 8481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:49.396034 8465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:50:49.396240 8465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "c1949806c8084d2fb41582e9dfdc7d8f"
format_stamp: "Formatted at 2026-09-25 02:50:49 on dist-test-slave-g1xh"
server_key: "0ff3bd0b2e6c2106774d5fb5c8c0aa9c"
server_key_iv: "aa9d6c29769a0ac34a629e3c9d0d0de5"
server_key_version: "encryptionkey@0"
I20260925 02:50:49.396478 8465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:49.418016 8465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:49.422394 8465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:49.422742 8465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:49.442077 8465 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:45023
I20260925 02:50:49.442114 8532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:45023 every 8 connection(s)
I20260925 02:50:49.443521 8465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:49.447263 8533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:49.453702 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 8465
I20260925 02:50:49.453953 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:49.454614 2737 external_mini_cluster.cc:1428] Setting key 25d9972104460b2c5d67759fe2ea80b6
I20260925 02:50:49.458598 8533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f: Bootstrap starting.
I20260925 02:50:49.462565 8533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:49.463997 8533 log.cc:824] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:49.467299 8533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f: No bootstrap required, opened a new log
Sep 25 02:50:49 dist-test-slave-g1xh krb5kdc[8452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:49.477577 8533 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1949806c8084d2fb41582e9dfdc7d8f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45023 } }
I20260925 02:50:49.478057 8533 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:49.478183 8533 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1949806c8084d2fb41582e9dfdc7d8f, State: Initialized, Role: FOLLOWER
I20260925 02:50:49.478883 8533 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1949806c8084d2fb41582e9dfdc7d8f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45023 } }
I20260925 02:50:49.479252 8533 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:49.479439 8533 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:49.479656 8533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:49.480618 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:49.456853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50390 (local address 127.2.172.126:45023)
0925 02:50:49.457680 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:49.457695 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:49.457747 (+ 52us) server_negotiation.cc:407] Connection header received
0925 02:50:49.460277 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:49.460308 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:49.460867 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:49.461312 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:49.462604 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:49.464746 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:49.466242 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:49.467254 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:49.467488 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:49.471655 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:49.471668 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:49.471688 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:49.471752 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:49.475979 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:49.476674 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:49.476683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:49.476693 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:49.476909 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:49.477271 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:49.477276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:49.477279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:49.478011 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:49.478321 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:49.478815 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 02:50:49.479277 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":354,"threads_started":1}
I20260925 02:50:49.482878 8533 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1949806c8084d2fb41582e9dfdc7d8f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45023 } }
I20260925 02:50:49.483418 8533 leader_election.cc:302] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1949806c8084d2fb41582e9dfdc7d8f; no voters:
I20260925 02:50:49.484171 8533 leader_election.cc:288] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:49.484277 8538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:49.485100 8538 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [term 1 LEADER]: Becoming Leader. State: Replica: c1949806c8084d2fb41582e9dfdc7d8f, State: Running, Role: LEADER
I20260925 02:50:49.485672 8538 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1949806c8084d2fb41582e9dfdc7d8f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45023 } }
I20260925 02:50:49.486335 8533 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:49.488701 8539 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1949806c8084d2fb41582e9dfdc7d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1949806c8084d2fb41582e9dfdc7d8f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45023 } } }
I20260925 02:50:49.489156 8539 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:49.489353 8540 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1949806c8084d2fb41582e9dfdc7d8f. Latest consensus state: current_term: 1 leader_uuid: "c1949806c8084d2fb41582e9dfdc7d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1949806c8084d2fb41582e9dfdc7d8f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 45023 } } }
I20260925 02:50:49.489717 8540 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:49.491643 8545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:49.497664 8545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:49.507782 8545 catalog_manager.cc:1347] Generated new cluster ID: 49927246987744c1aa42a05372a19a1f
I20260925 02:50:49.507925 8545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:49.533278 8545 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:49.534695 8545 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:49.541766 8545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1949806c8084d2fb41582e9dfdc7d8f: Generated new TSK 0
I20260925 02:50:49.542662 8545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:50:49.619239 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:45023
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:49.963238 8560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:49.963603 8560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:49.963724 8560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:49.971319 8560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:49.971444 8560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:49.971642 8560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:50:49.982550 8560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:45023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:49.984002 8560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:49.985353 8560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:49.999838 8565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:49.999851 8566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:49.999992 8568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:50.001188 8560 server_base.cc:1034] running on GCE node
I20260925 02:50:50.001853 8560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:50.002856 8560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:50.004128 8560 hybrid_clock.cc:648] HybridClock initialized: now 1790304650004057 us; error 64 us; skew 500 ppm
I20260925 02:50:50.004527 8560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:50 dist-test-slave-g1xh krb5kdc[8452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304650, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:50.008054 8560 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:50:50.009740 8560 webserver.cc:460] Webserver started at http://127.2.172.65:34421/ using document root <none> and password file <none>
I20260925 02:50:50.010468 8560 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:50.010576 8560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:50.010840 8560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:50.014056 8560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "5281f5db29ca46e0b3649895978d38e9"
format_stamp: "Formatted at 2026-09-25 02:50:50 on dist-test-slave-g1xh"
server_key: "36b7f33cafefa4be879f6d0a08aca889"
server_key_iv: "f4d7f055129e1b9cad3b3259dad57d18"
server_key_version: "encryptionkey@0"
I20260925 02:50:50.014878 8560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "5281f5db29ca46e0b3649895978d38e9"
format_stamp: "Formatted at 2026-09-25 02:50:50 on dist-test-slave-g1xh"
server_key: "36b7f33cafefa4be879f6d0a08aca889"
server_key_iv: "f4d7f055129e1b9cad3b3259dad57d18"
server_key_version: "encryptionkey@0"
I20260925 02:50:50.020895 8560 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260925 02:50:50.024881 8575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:50.026475 8560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:50:50.026688 8560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "5281f5db29ca46e0b3649895978d38e9"
format_stamp: "Formatted at 2026-09-25 02:50:50 on dist-test-slave-g1xh"
server_key: "36b7f33cafefa4be879f6d0a08aca889"
server_key_iv: "f4d7f055129e1b9cad3b3259dad57d18"
server_key_version: "encryptionkey@0"
I20260925 02:50:50.026916 8560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:50.045843 8560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:50.049938 8560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:50.050244 8560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:50.051210 8560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:50.052626 8560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:50.052706 8560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:50.052881 8560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:50.052939 8560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:50.086663 8560 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:39733
I20260925 02:50:50.086683 8690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:39733 every 8 connection(s)
I20260925 02:50:50.088189 8560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:50.094866 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 8560
I20260925 02:50:50.095099 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:50.095767 2737 external_mini_cluster.cc:1428] Setting key 1c9dd91685c58e94adb54720228682a3
Sep 25 02:50:50 dist-test-slave-g1xh krb5kdc[8452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304650, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:50.110387 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:50.090857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:40953 (local address 127.2.172.126:45023)
0925 02:50:50.091430 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:50.091437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:50.092494 (+ 1057us) server_negotiation.cc:407] Connection header received
0925 02:50:50.094345 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:50.094349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:50.094419 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:50.094558 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:50.096395 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:50.097257 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:50.098975 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:50.099765 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:50.099929 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:50.103737 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:50.103746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:50.103753 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:50.103799 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:50.106995 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:50.108008 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:50.108012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:50.108015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:50.108131 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:50.108631 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:50.108634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:50.108636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:50.108892 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:50.109010 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:50.109952 (+ 942us) server_negotiation.cc:299] Negotiation successful
0925 02:50:50.110129 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
I20260925 02:50:50.112077 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:50.091431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:45023 (local address 127.2.172.65:40953)
0925 02:50:50.092322 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:50.092383 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:50.094073 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:50.094837 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:50.094846 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:50.095492 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:50.096186 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:50.096204 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:50.097438 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:50.097447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:50.098815 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:50.098828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:50.099998 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:50.100003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:50.100274 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:50.101012 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:50.101039 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:50.103471 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:50.107146 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:50.107159 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:50.107181 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:50.107882 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:50.108248 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:50.108255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:50.108262 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:50.108520 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:50.109028 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:50.109039 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:50.109529 (+ 490us) client_negotiation.cc:769] Sending connection context
0925 02:50:50.109875 (+ 346us) client_negotiation.cc:240] Negotiation successful
0925 02:50:50.110487 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":355,"threads_started":1}
I20260925 02:50:50.113672 8691 heartbeater.cc:347] Connected to a master server at 127.2.172.126:45023
I20260925 02:50:50.113986 8691 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:50.114746 8691 heartbeater.cc:511] Master 127.2.172.126:45023 requested a full tablet report, sending...
I20260925 02:50:50.116981 8498 ts_manager.cc:194] Registered new tserver with Master: 5281f5db29ca46e0b3649895978d38e9 (127.2.172.65:39733)
I20260925 02:50:50.118431 8498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:40953
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:50:50.148042 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:45023
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:50.493299 8697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:50.493659 8697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:50.493839 8697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:50.501683 8697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:50.501854 8697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:50.502041 8697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:50:50.512111 8697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:45023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:50.513592 8697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:50.514935 8697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:50.528578 8703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:50.528666 8705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:50.528625 8702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:50.529903 8697 server_base.cc:1034] running on GCE node
I20260925 02:50:50.530579 8697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:50.531697 8697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:50.532960 8697 hybrid_clock.cc:648] HybridClock initialized: now 1790304650532918 us; error 40 us; skew 500 ppm
I20260925 02:50:50.533306 8697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:50 dist-test-slave-g1xh krb5kdc[8452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304650, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:50.536825 8697 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:50:50.538591 8697 webserver.cc:460] Webserver started at http://127.2.172.66:41861/ using document root <none> and password file <none>
I20260925 02:50:50.539399 8697 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:50.539639 8697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:50.540002 8697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:50.543303 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "8f19dd545652445088b033871db50ab7"
format_stamp: "Formatted at 2026-09-25 02:50:50 on dist-test-slave-g1xh"
server_key: "32fbdab3cc3c11efc62948afdd37e520"
server_key_iv: "56e4dd46b47ca0b878aa734a389cce4d"
server_key_version: "encryptionkey@0"
I20260925 02:50:50.544299 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "8f19dd545652445088b033871db50ab7"
format_stamp: "Formatted at 2026-09-25 02:50:50 on dist-test-slave-g1xh"
server_key: "32fbdab3cc3c11efc62948afdd37e520"
server_key_iv: "56e4dd46b47ca0b878aa734a389cce4d"
server_key_version: "encryptionkey@0"
I20260925 02:50:50.552081 8697 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260925 02:50:50.556092 8712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:50.557575 8697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:50:50.557776 8697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "8f19dd545652445088b033871db50ab7"
format_stamp: "Formatted at 2026-09-25 02:50:50 on dist-test-slave-g1xh"
server_key: "32fbdab3cc3c11efc62948afdd37e520"
server_key_iv: "56e4dd46b47ca0b878aa734a389cce4d"
server_key_version: "encryptionkey@0"
I20260925 02:50:50.558012 8697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:50.579213 8697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:50.584056 8697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:50.584399 8697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:50.585436 8697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:50.587046 8697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:50.587167 8697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:50.587271 8697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:50.587320 8697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:50.621794 8697 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:34619
I20260925 02:50:50.621840 8825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:34619 every 8 connection(s)
I20260925 02:50:50.623075 8697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:50.630163 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 8697
I20260925 02:50:50.630374 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:50.630991 2737 external_mini_cluster.cc:1428] Setting key 18d1f099e6163bc5ec036285f71dcf0a
Sep 25 02:50:50 dist-test-slave-g1xh krb5kdc[8452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304650, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:50.644627 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:50.625686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:33623 (local address 127.2.172.126:45023)
0925 02:50:50.626165 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:50.626171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:50.627086 (+ 915us) server_negotiation.cc:407] Connection header received
0925 02:50:50.628885 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:50.628890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:50.628962 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:50.629061 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:50.630842 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:50.631744 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:50.633779 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:50.634620 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:50.634777 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:50.638166 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:50.638175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:50.638183 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:50.638238 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:50.641183 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:50.642159 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:50.642163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:50.642166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:50.642359 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:50.642830 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:50.642833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:50.642835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:50.643091 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:50.643200 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:50.644143 (+ 943us) server_negotiation.cc:299] Negotiation successful
0925 02:50:50.644350 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":218,"threads_started":1}
I20260925 02:50:50.646139 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:50.626187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:45023 (local address 127.2.172.66:33623)
0925 02:50:50.626921 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:50.626966 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:50.628610 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:50.629264 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:50.629272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:50.629823 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:50.630652 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:50.630677 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:50.631946 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:50.631950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:50.633632 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:50.633644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:50.634834 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:50.634839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:50.635050 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:50.635773 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:50.635800 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:50.637923 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:50.641342 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:50.641354 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:50.641375 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:50.642032 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:50.642471 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:50.642478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:50.642485 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:50.642724 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:50.643218 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:50.643229 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:50.643741 (+ 512us) client_negotiation.cc:769] Sending connection context
0925 02:50:50.644066 (+ 325us) client_negotiation.cc:240] Negotiation successful
0925 02:50:50.644646 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":275,"threads_started":1}
I20260925 02:50:50.647688 8826 heartbeater.cc:347] Connected to a master server at 127.2.172.126:45023
I20260925 02:50:50.648020 8826 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:50.648775 8826 heartbeater.cc:511] Master 127.2.172.126:45023 requested a full tablet report, sending...
I20260925 02:50:50.650364 8498 ts_manager.cc:194] Registered new tserver with Master: 8f19dd545652445088b033871db50ab7 (127.2.172.66:34619)
I20260925 02:50:50.651239 8498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:33623
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:50:50.682600 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:45023
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:51.017815 8833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:51.018139 8833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:51.018237 8833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:51.025862 8833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:51.025995 8833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:51.026109 8833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:50:51.036651 8833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:45023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:51.037935 8833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:51.039178 8833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:51.052934 8838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:51.052855 8839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:51.052814 8841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:51.054445 8833 server_base.cc:1034] running on GCE node
I20260925 02:50:51.055070 8833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:51.056198 8833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:51.057435 8833 hybrid_clock.cc:648] HybridClock initialized: now 1790304651057377 us; error 41 us; skew 500 ppm
I20260925 02:50:51.057785 8833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:51.065271 8833 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:50:51.066766 8833 webserver.cc:460] Webserver started at http://127.2.172.67:46479/ using document root <none> and password file <none>
I20260925 02:50:51.067482 8833 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:51.067677 8833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:51.068004 8833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:51.070912 8833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "5d9abc047efc4a86b37f6eca8f8e99e6"
format_stamp: "Formatted at 2026-09-25 02:50:51 on dist-test-slave-g1xh"
server_key: "0e23c3ec235722a3c3cde850b908ba53"
server_key_iv: "a5b2ef71a672a420e78ea941ba813ffb"
server_key_version: "encryptionkey@0"
I20260925 02:50:51.071775 8833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "5d9abc047efc4a86b37f6eca8f8e99e6"
format_stamp: "Formatted at 2026-09-25 02:50:51 on dist-test-slave-g1xh"
server_key: "0e23c3ec235722a3c3cde850b908ba53"
server_key_iv: "a5b2ef71a672a420e78ea941ba813ffb"
server_key_version: "encryptionkey@0"
I20260925 02:50:51.077147 8833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 02:50:51.080791 8848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:51.082217 8833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:50:51.082427 8833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "5d9abc047efc4a86b37f6eca8f8e99e6"
format_stamp: "Formatted at 2026-09-25 02:50:51 on dist-test-slave-g1xh"
server_key: "0e23c3ec235722a3c3cde850b908ba53"
server_key_iv: "a5b2ef71a672a420e78ea941ba813ffb"
server_key_version: "encryptionkey@0"
I20260925 02:50:51.082654 8833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:51.104765 8833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:51.109272 8833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:51.109607 8833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:51.110648 8833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:51.112094 8833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:51.112196 8833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:51.112349 8833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:51.112423 8833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:51.121428 8691 heartbeater.cc:503] Master 127.2.172.126:45023 was elected leader, sending a full tablet report...
I20260925 02:50:51.146582 8833 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:37987
I20260925 02:50:51.146615 8961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:37987 every 8 connection(s)
I20260925 02:50:51.147895 8833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:51.155478 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 8833
I20260925 02:50:51.155706 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:51.156235 2737 external_mini_cluster.cc:1428] Setting key 2409e9c6097d0889e9e7c27a93229079
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:50:51.168152 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:51.150342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:54347 (local address 127.2.172.126:45023)
0925 02:50:51.150762 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:51.150767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:51.151804 (+ 1037us) server_negotiation.cc:407] Connection header received
0925 02:50:51.153509 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:51.153513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:51.153604 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:51.153702 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:51.155164 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.156364 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.158146 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.158632 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.158731 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:51.162564 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:51.162569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:51.162573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:51.162613 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:51.165309 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.166034 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.166038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.166041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.166168 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.166584 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.166588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.166590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.166876 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:51.167008 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:51.167672 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 02:50:51.167845 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":198,"threads_started":1}
I20260925 02:50:51.169122 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:51.150907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:45023 (local address 127.2.172.67:54347)
0925 02:50:51.151655 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:51.151693 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:51.153246 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:51.153912 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:51.153919 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:51.154419 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:51.154977 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:51.154997 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.156505 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.156509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:51.157999 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:51.158015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.158859 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.158866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:51.159185 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:51.159965 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:51.159992 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:51.162300 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:51.165453 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:51.165460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:51.165474 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:51.165885 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:51.166282 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:51.166285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:51.166287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:51.166468 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:51.166993 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:51.166999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:51.167303 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 02:50:51.167527 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 02:50:51.167984 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":270,"threads_started":1}
I20260925 02:50:51.170828 8962 heartbeater.cc:347] Connected to a master server at 127.2.172.126:45023
I20260925 02:50:51.171164 8962 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:51.172106 8962 heartbeater.cc:511] Master 127.2.172.126:45023 requested a full tablet report, sending...
I20260925 02:50:51.173540 8498 ts_manager.cc:194] Registered new tserver with Master: 5d9abc047efc4a86b37f6eca8f8e99e6 (127.2.172.67:37987)
I20260925 02:50:51.174350 8498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:54347
I20260925 02:50:51.184656 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/test-admin.keytab.
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:51.215006 2737 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 02:50:51.220582 8970 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:50:51.217924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.2.172.65:39733)
0925 02:50:51.218625 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:51.218633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:51.218661 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:50:51.218721 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:51.218727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:51.218944 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:51.219079 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:51.220143 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44630: BlockingRecv error: recv got EOF from 127.0.0.1:44630 (error 108)
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":430,"threads_started":1}
W20260925 02:50:51.221097 2737 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:51.223228 2737 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:51.224869 2737 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:51.242693 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:51.226864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.2.172.65:39733)
0925 02:50:51.227099 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:51.227106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:51.227437 (+ 331us) server_negotiation.cc:407] Connection header received
0925 02:50:51.227638 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:51.227642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:51.227718 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:51.227815 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:51.228899 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.229924 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.231212 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.231970 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.232074 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:51.236252 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:51.236271 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:51.236289 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:51.236333 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:51.239331 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.240258 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.240268 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.240290 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.240489 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.240985 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.240989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.240992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.241564 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:51.241822 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:51.242181 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 02:50:51.242424 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 02:50:51.249192 8970 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:50:51.247462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.2.172.65:39733)
0925 02:50:51.247706 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:51.247719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:51.247851 (+ 132us) server_negotiation.cc:407] Connection header received
0925 02:50:51.248010 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:51.248014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:51.248088 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:51.248170 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:51.248916 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44642: BlockingRecv error: recv got EOF from 127.0.0.1:44642 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260925 02:50:51.249399 2737 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:51.251233 2737 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:51.252820 2737 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:51.268359 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:51.254475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.2.172.65:39733)
0925 02:50:51.254719 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:51.254726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:51.255027 (+ 301us) server_negotiation.cc:407] Connection header received
0925 02:50:51.255162 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:51.255166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:51.255231 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:51.255316 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:51.256427 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.257124 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.258402 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.258838 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.258948 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:51.262670 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:51.262675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:51.262678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:51.262731 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:51.265475 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.266232 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.266237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.266239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.266384 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.266772 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.266776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.266778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.267224 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:51.267428 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:51.267810 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 02:50:51.268048 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 02:50:51.273217 8970 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:50:51.271276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.2.172.65:39733)
0925 02:50:51.271500 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:51.271512 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:51.271795 (+ 283us) server_negotiation.cc:407] Connection header received
0925 02:50:51.271994 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:51.271998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:51.272071 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:51.272177 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:51.272989 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44670: BlockingRecv error: recv got EOF from 127.0.0.1:44670 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 02:50:51.273598 2737 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304596269475-2737-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:51.275218 2737 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:51.276695 2737 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:50:51.292376 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:51.278161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.2.172.65:39733)
0925 02:50:51.278430 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:51.278439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:51.279697 (+ 1258us) server_negotiation.cc:407] Connection header received
0925 02:50:51.279821 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:51.279825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:51.279892 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:51.279979 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:51.280947 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.281649 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.282827 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.283228 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.283328 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:51.287099 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:51.287105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:51.287108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:51.287147 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:51.289762 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.290444 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.290448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.290451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.290611 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.290932 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.290935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.290937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.291196 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:51.291487 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:51.291754 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 02:50:51.292034 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:50:51.294034 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 8560
I20260925 02:50:51.306277 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:51.294705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.2.172.65:39733)
0925 02:50:51.294939 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:51.294945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:51.295183 (+ 238us) server_negotiation.cc:407] Connection header received
0925 02:50:51.295387 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:51.295392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:51.295460 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:51.295619 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:51.296556 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.297237 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.298480 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.298913 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.299051 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:51.301210 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:51.301215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:51.301218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:51.301252 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:51.303615 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.304304 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.304310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.304314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.304462 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.304847 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.304852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.304856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.305137 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:51.305413 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:51.305725 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 02:50:51.305952 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:50:51.306852 8685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:51.390162 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 8560
I20260925 02:50:51.408607 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 8697
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:50:51.424111 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:51.409550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.2.172.66:34619)
0925 02:50:51.410115 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:51.410122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:51.410147 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:50:51.410207 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:51.410213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:51.410453 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:51.410591 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:51.411804 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.412890 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.414094 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.414651 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.414749 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:51.418568 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:51.418581 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:51.418595 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:51.418635 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:51.421563 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.422180 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.422186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.422189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.422305 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.422651 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.422654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.422656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.422970 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:51.423204 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:51.423479 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 02:50:51.423782 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":291,"threads_started":1}
I20260925 02:50:51.425361 8821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:51.515496 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 8697
I20260925 02:50:51.533342 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 8833
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:50:51.551062 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:51.534397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.2.172.67:37987)
0925 02:50:51.535060 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:51.535067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:51.535098 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:50:51.535158 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:51.535164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:51.535397 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:51.535540 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:51.536673 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.537815 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.539131 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:51.539831 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:51.539944 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:51.544433 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:51.544450 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:51.544467 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:51.544509 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:51.547772 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.548614 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.548624 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.548628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.548779 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:51.549222 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:51.549227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:51.549231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:51.549619 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:51.549971 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:51.550290 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 02:50:51.550579 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":346,"threads_started":1}
I20260925 02:50:51.552872 8957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:51.639469 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 8833
I20260925 02:50:51.657078 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 8465
I20260925 02:50:51.658301 8528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:51.784297 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 8465
2026-09-25T02:50:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4113 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8991](info): set up 2 sockets
Sep 25 02:50:51 dist-test-slave-g1xh krb5kdc[8991](info): commencing operation
krb5kdc: starting...
W20260925 02:50:52.947933 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:50:53 dist-test-slave-g1xh krb5kdc[8991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.126.keytab.
Entry for principal oryx/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.126.keytab.
I20260925 02:50:53.185330 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:36101
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.126.keytab
--principal=oryx/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:53.516060 9004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:53.516440 9004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:53.516623 9004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:53.524271 9004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:53.524453 9004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:53.524541 9004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:53.524652 9004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:53.524730 9004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:53.535418 9004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:36101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.126.keytab
--principal=oryx/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:53.537036 9004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:53.538479 9004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:53.553373 9009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:53.553663 9010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:53.554783 9012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:53.555774 9004 server_base.cc:1034] running on GCE node
I20260925 02:50:53.557152 9004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:53.559131 9004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:53.560451 9004 hybrid_clock.cc:648] HybridClock initialized: now 1790304653560381 us; error 53 us; skew 500 ppm
I20260925 02:50:53.560909 9004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:53 dist-test-slave-g1xh krb5kdc[8991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304653, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:53.572531 9004 init.cc:377] Logged in from keytab as oryx/127.2.172.126@KRBTEST.COM (short username oryx)
I20260925 02:50:53.575273 9004 webserver.cc:460] Webserver started at http://127.2.172.126:38593/ using document root <none> and password file <none>
I20260925 02:50:53.576828 9004 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:53.577030 9004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:53.577499 9004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:53.583061 9004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "e0fda7481e654efa9d6911ae0af27ec6"
format_stamp: "Formatted at 2026-09-25 02:50:53 on dist-test-slave-g1xh"
server_key: "e68067173c13ebc2b4b8bea974dc349d"
server_key_iv: "c26618b51118a51ce0924920f0564126"
server_key_version: "encryptionkey@0"
I20260925 02:50:53.584352 9004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "e0fda7481e654efa9d6911ae0af27ec6"
format_stamp: "Formatted at 2026-09-25 02:50:53 on dist-test-slave-g1xh"
server_key: "e68067173c13ebc2b4b8bea974dc349d"
server_key_iv: "c26618b51118a51ce0924920f0564126"
server_key_version: "encryptionkey@0"
I20260925 02:50:53.592129 9004 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260925 02:50:53.596338 9019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:53.597958 9004 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:50:53.598166 9004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "e0fda7481e654efa9d6911ae0af27ec6"
format_stamp: "Formatted at 2026-09-25 02:50:53 on dist-test-slave-g1xh"
server_key: "e68067173c13ebc2b4b8bea974dc349d"
server_key_iv: "c26618b51118a51ce0924920f0564126"
server_key_version: "encryptionkey@0"
I20260925 02:50:53.598469 9004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:53.632282 9004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:53.636677 9004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:53.637019 9004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:53.660411 9004 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:36101
I20260925 02:50:53.660461 9070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:36101 every 8 connection(s)
I20260925 02:50:53.662674 9004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:53.668215 9071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:53.668879 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 9004
I20260925 02:50:53.669116 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:50:53.670038 2737 external_mini_cluster.cc:1428] Setting key ccaa4d3d1639c1e89e9294835ef61eb7
I20260925 02:50:53.682765 9071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6: Bootstrap starting.
Sep 25 02:50:53 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:50:53.686285 9071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:53.687759 9071 log.cc:824] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:53.691401 9071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6: No bootstrap required, opened a new log
I20260925 02:50:53.695645 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:53.672458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.2.172.126:36101)
0925 02:50:53.673345 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:53.673359 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:53.673408 (+ 49us) server_negotiation.cc:407] Connection header received
0925 02:50:53.675477 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:53.675505 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:53.676096 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:53.676574 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:53.678193 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:53.679671 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:53.681097 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:53.681854 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:53.682112 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:53.686435 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:53.686470 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:53.686494 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:53.686553 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:53.690685 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:53.691500 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:53.691507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:53.691520 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:53.691793 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:53.692202 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:53.692207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:53.692212 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:53.692766 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:53.693119 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:53.693622 (+ 503us) server_negotiation.cc:299] Negotiation successful
0925 02:50:53.694363 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":436,"threads_started":1}
I20260925 02:50:53.700802 9071 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } }
I20260925 02:50:53.701216 9071 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:53.701324 9071 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fda7481e654efa9d6911ae0af27ec6, State: Initialized, Role: FOLLOWER
I20260925 02:50:53.701886 9071 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } }
I20260925 02:50:53.702142 9071 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:53.702289 9071 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:53.702462 9071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:53.705013 9071 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } }
I20260925 02:50:53.705634 9071 leader_election.cc:302] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0fda7481e654efa9d6911ae0af27ec6; no voters:
I20260925 02:50:53.706276 9071 leader_election.cc:288] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:53.706425 9076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:53.707150 9076 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 1 LEADER]: Becoming Leader. State: Replica: e0fda7481e654efa9d6911ae0af27ec6, State: Running, Role: LEADER
I20260925 02:50:53.707798 9076 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } }
I20260925 02:50:53.708608 9071 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:53.710899 9077 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } }
I20260925 02:50:53.711302 9077 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:53.711002 9078 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0fda7481e654efa9d6911ae0af27ec6. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } }
I20260925 02:50:53.711463 9078 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:53.715145 9081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:53.721611 9081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:53.731016 9081 catalog_manager.cc:1347] Generated new cluster ID: d9373e52acd046568f4e4682f8104893
I20260925 02:50:53.731164 9081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:53.758888 9081 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:53.760591 9081 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:53.772257 9081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6: Generated new TSK 0
I20260925 02:50:53.773456 9081 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab.
Entry for principal oryx/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab.
I20260925 02:50:53.840219 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:36101
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab
--principal=oryx/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:54.194085 9097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:54.194408 9097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:54.194623 9097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:54.202172 9097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:54.202340 9097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:54.202510 9097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:50:54.213392 9097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab
--principal=oryx/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:36101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:54.214969 9097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:54.216676 9097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:54.230712 9102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:54.230638 9105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:54.230664 9103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:54.231521 9097 server_base.cc:1034] running on GCE node
I20260925 02:50:54.232183 9097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:54.233191 9097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:54.234404 9097 hybrid_clock.cc:648] HybridClock initialized: now 1790304654234332 us; error 59 us; skew 500 ppm
I20260925 02:50:54.234743 9097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:54 dist-test-slave-g1xh krb5kdc[8991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304654, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:54.238042 9097 init.cc:377] Logged in from keytab as oryx/127.2.172.65@KRBTEST.COM (short username oryx)
I20260925 02:50:54.239720 9097 webserver.cc:460] Webserver started at http://127.2.172.65:43923/ using document root <none> and password file <none>
I20260925 02:50:54.240527 9097 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:54.240686 9097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:54.241014 9097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:54.244195 9097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "30ab128117884673903a3c39d748b7a2"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-g1xh"
server_key: "ee2aae5ab861084a57279c700e314ae2"
server_key_iv: "ea7910d83e25fa867b05d5072896ebfb"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.245014 9097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "30ab128117884673903a3c39d748b7a2"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-g1xh"
server_key: "ee2aae5ab861084a57279c700e314ae2"
server_key_iv: "ea7910d83e25fa867b05d5072896ebfb"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.250541 9097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:50:54.254370 9112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.256283 9097 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:50:54.256512 9097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "30ab128117884673903a3c39d748b7a2"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-g1xh"
server_key: "ee2aae5ab861084a57279c700e314ae2"
server_key_iv: "ea7910d83e25fa867b05d5072896ebfb"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.256762 9097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:54.278282 9097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:54.282845 9097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:54.283155 9097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:54.284302 9097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:54.285732 9097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:54.285838 9097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.286005 9097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:54.286079 9097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.321583 9097 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:39097
I20260925 02:50:54.321640 9226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:39097 every 8 connection(s)
I20260925 02:50:54.322962 9097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:54.326772 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 9097
I20260925 02:50:54.327025 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:54.327793 2737 external_mini_cluster.cc:1428] Setting key c4008470924b22607d0db65a241b60c8
Sep 25 02:50:54 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304654, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:50:54.344595 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.325723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:40339 (local address 127.2.172.126:36101)
0925 02:50:54.326289 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.326294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.329750 (+ 3456us) server_negotiation.cc:407] Connection header received
0925 02:50:54.329801 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.329805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.329865 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.329984 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:54.331592 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.332749 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.334306 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.334822 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.334964 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.338606 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:54.338611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:54.338615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:54.338649 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:54.341307 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.342082 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.342088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.342097 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.342216 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.342718 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.342723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.342729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.343012 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:54.343172 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.343975 (+ 803us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.344209 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":274,"threads_started":1}
I20260925 02:50:54.345682 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.326207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.65:40339)
0925 02:50:54.327338 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:54.327395 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:54.329427 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:54.330242 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:54.330251 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:54.330777 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:54.331346 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.331364 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.332910 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.332915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.334137 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.334149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.335078 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.335087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.335332 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.336132 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:54.336160 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:54.338356 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:54.341466 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:54.341476 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:54.341496 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:54.341942 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:54.342387 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:54.342393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:54.342396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:54.342599 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:54.343146 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:54.343154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:54.343512 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 02:50:54.343873 (+ 361us) client_negotiation.cc:240] Negotiation successful
0925 02:50:54.344430 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":389,"threads_started":1}
I20260925 02:50:54.347527 9227 heartbeater.cc:347] Connected to a master server at 127.2.172.126:36101
I20260925 02:50:54.347873 9227 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:54.348598 9227 heartbeater.cc:511] Master 127.2.172.126:36101 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.2.172.66@KRBTEST.COM; defaulting to no policy
I20260925 02:50:54.350963 9036 ts_manager.cc:194] Registered new tserver with Master: 30ab128117884673903a3c39d748b7a2 (127.2.172.65:39097)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.66@KRBTEST.COM" created.
I20260925 02:50:54.352824 9036 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:40339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab.
Entry for principal oryx/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab.
I20260925 02:50:54.379875 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:36101
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab
--principal=oryx/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:54.721796 9233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:54.722088 9233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:54.722190 9233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:54.729449 9233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:54.729574 9233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:54.729687 9233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:50:54.739719 9233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab
--principal=oryx/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:36101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:54.741050 9233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:54.742466 9233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:54.756124 9241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:54.756036 9238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:54.756166 9239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:54.756879 9233 server_base.cc:1034] running on GCE node
I20260925 02:50:54.757613 9233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:54.758697 9233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:54.759904 9233 hybrid_clock.cc:648] HybridClock initialized: now 1790304654759839 us; error 63 us; skew 500 ppm
I20260925 02:50:54.760254 9233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:54 dist-test-slave-g1xh krb5kdc[8991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304654, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:54.764029 9233 init.cc:377] Logged in from keytab as oryx/127.2.172.66@KRBTEST.COM (short username oryx)
I20260925 02:50:54.765673 9233 webserver.cc:460] Webserver started at http://127.2.172.66:36047/ using document root <none> and password file <none>
I20260925 02:50:54.766439 9233 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:54.766595 9233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:54.766937 9233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:54.770377 9233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "79fab9fd748347559e7d7710682b465a"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-g1xh"
server_key: "0ade64b497265de5fddbdec14960f034"
server_key_iv: "1fea47ba5a76342eb15c62a6db4c0111"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.771237 9233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "79fab9fd748347559e7d7710682b465a"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-g1xh"
server_key: "0ade64b497265de5fddbdec14960f034"
server_key_iv: "1fea47ba5a76342eb15c62a6db4c0111"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.776656 9233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 02:50:54.780206 9249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.781702 9233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:50:54.781936 9233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "79fab9fd748347559e7d7710682b465a"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-g1xh"
server_key: "0ade64b497265de5fddbdec14960f034"
server_key_iv: "1fea47ba5a76342eb15c62a6db4c0111"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.782197 9233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:54.800716 9233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:54.805392 9233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:54.805694 9233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:54.806675 9233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:54.808065 9233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:54.808184 9233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.808449 9233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:54.808537 9233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.849364 9233 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:33489
I20260925 02:50:54.849413 9362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:33489 every 8 connection(s)
I20260925 02:50:54.850690 9233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:54.853631 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 9233
I20260925 02:50:54.853830 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:54.854404 2737 external_mini_cluster.cc:1428] Setting key 20f44e9ebd0c77cfd7f1f4eb634ada1e
Sep 25 02:50:54 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304654, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:50:54.872905 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.853380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:57057 (local address 127.2.172.126:36101)
0925 02:50:54.853866 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.853891 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.855135 (+ 1244us) server_negotiation.cc:407] Connection header received
0925 02:50:54.857216 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.857222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.857339 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.857482 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:54.859002 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.860285 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.862037 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.862825 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.862983 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.867087 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:54.867093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:54.867096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:54.867131 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:54.869779 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.870507 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.870515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.870520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.870654 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.871092 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.871098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.871101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.871374 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:54.871534 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.872249 (+ 715us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.872509 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":265,"threads_started":1}
I20260925 02:50:54.873914 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.854103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.66:57057)
0925 02:50:54.854932 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:54.854976 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:54.856945 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:54.857673 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:54.857681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:54.858225 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:54.858825 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.858842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.860490 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.860499 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.861889 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.861908 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.863071 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.863076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.863307 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.864125 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:54.864165 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:54.866837 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:54.869931 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:54.869939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:54.869957 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:54.870380 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:54.870768 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:54.870773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:54.870776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:54.870976 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:54.871505 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:54.871513 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:54.871899 (+ 386us) client_negotiation.cc:769] Sending connection context
0925 02:50:54.872180 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 02:50:54.872752 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":363,"threads_started":1}
I20260925 02:50:54.875461 9363 heartbeater.cc:347] Connected to a master server at 127.2.172.126:36101
I20260925 02:50:54.875829 9363 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.2.172.67@KRBTEST.COM; defaulting to no policy
I20260925 02:50:54.876569 9363 heartbeater.cc:511] Master 127.2.172.126:36101 requested a full tablet report, sending...
I20260925 02:50:54.878177 9036 ts_manager.cc:194] Registered new tserver with Master: 79fab9fd748347559e7d7710682b465a (127.2.172.66:33489)
I20260925 02:50:54.879079 9036 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:57057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab.
Entry for principal oryx/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab.
I20260925 02:50:54.907935 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:36101
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab
--principal=oryx/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:55.293699 9369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:55.294062 9369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:55.294231 9369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:55.302333 9369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:55.302479 9369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:55.302675 9369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:50:55.314584 9369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab
--principal=oryx/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:36101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:55.316591 9369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:55.318349 9369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:55.336194 9375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:55.336146 9377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:55.336114 9374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:55.337428 9369 server_base.cc:1034] running on GCE node
I20260925 02:50:55.338183 9369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:55.339282 9369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:55.340516 9369 hybrid_clock.cc:648] HybridClock initialized: now 1790304655340448 us; error 49 us; skew 500 ppm
I20260925 02:50:55.340867 9369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:55 dist-test-slave-g1xh krb5kdc[8991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304655, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:55.344599 9369 init.cc:377] Logged in from keytab as oryx/127.2.172.67@KRBTEST.COM (short username oryx)
I20260925 02:50:55.346302 9369 webserver.cc:460] Webserver started at http://127.2.172.67:40883/ using document root <none> and password file <none>
I20260925 02:50:55.347182 9369 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:55.347322 9369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:55.347760 9369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:55.351415 9369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "080f175629c54c96b845358d2413c7f7"
format_stamp: "Formatted at 2026-09-25 02:50:55 on dist-test-slave-g1xh"
server_key: "f6bf95e9072738f87b161536fd3d7c64"
server_key_iv: "daacfaf74f66d310edd3dd3394b2bed3"
server_key_version: "encryptionkey@0"
I20260925 02:50:55.352497 9369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "080f175629c54c96b845358d2413c7f7"
format_stamp: "Formatted at 2026-09-25 02:50:55 on dist-test-slave-g1xh"
server_key: "f6bf95e9072738f87b161536fd3d7c64"
server_key_iv: "daacfaf74f66d310edd3dd3394b2bed3"
server_key_version: "encryptionkey@0"
I20260925 02:50:55.356778 9227 heartbeater.cc:503] Master 127.2.172.126:36101 was elected leader, sending a full tablet report...
I20260925 02:50:55.359239 9369 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 02:50:55.363505 9384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:55.365430 9369 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:50:55.365609 9369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "080f175629c54c96b845358d2413c7f7"
format_stamp: "Formatted at 2026-09-25 02:50:55 on dist-test-slave-g1xh"
server_key: "f6bf95e9072738f87b161536fd3d7c64"
server_key_iv: "daacfaf74f66d310edd3dd3394b2bed3"
server_key_version: "encryptionkey@0"
I20260925 02:50:55.365818 9369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:55.385736 9369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:55.391368 9369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:55.391762 9369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:55.392988 9369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:55.394738 9369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:55.394820 9369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:55.394917 9369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:55.394968 9369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:55.434001 9369 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:40383
I20260925 02:50:55.434254 9497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:40383 every 8 connection(s)
I20260925 02:50:55.435761 9369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:55.444737 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 9369
I20260925 02:50:55.445014 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:55.445626 2737 external_mini_cluster.cc:1428] Setting key dc95bfc32d0d12d2513c3f1cd717564e
Sep 25 02:50:55 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304655, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:50:55.461874 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:55.438996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:56239 (local address 127.2.172.126:36101)
0925 02:50:55.439609 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:55.439618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:55.440744 (+ 1126us) server_negotiation.cc:407] Connection header received
0925 02:50:55.442893 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:55.442900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:55.443029 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:55.443166 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:55.445376 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:55.446532 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:55.448899 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:55.449599 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:55.449729 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:55.454085 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:55.454091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:55.454095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:55.454149 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:55.457330 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:55.458284 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:55.458289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:55.458292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:55.458438 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:55.458953 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:55.458959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:55.458963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:55.459347 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:55.459506 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:55.460955 (+ 1449us) server_negotiation.cc:299] Negotiation successful
0925 02:50:55.461264 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":298,"threads_started":1}
I20260925 02:50:55.463395 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:55.439697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.67:56239)
0925 02:50:55.440537 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:55.440593 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:55.442572 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:55.443451 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:55.443460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:55.444136 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:55.445147 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:55.445175 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:55.446711 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:55.446717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:55.448694 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:55.448708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:55.449916 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:55.449923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:55.450213 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:55.451084 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:55.451117 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:55.453774 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:55.457529 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:55.457540 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:55.457560 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:55.458130 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:55.458584 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:55.458588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:55.458590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:55.458804 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:55.459503 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:55.459512 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:55.460213 (+ 701us) client_negotiation.cc:769] Sending connection context
0925 02:50:55.460773 (+ 560us) client_negotiation.cc:240] Negotiation successful
0925 02:50:55.461534 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":335,"threads_started":1}
I20260925 02:50:55.465207 9498 heartbeater.cc:347] Connected to a master server at 127.2.172.126:36101
I20260925 02:50:55.465730 9498 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:55.466739 9498 heartbeater.cc:511] Master 127.2.172.126:36101 requested a full tablet report, sending...
I20260925 02:50:55.468914 9036 ts_manager.cc:194] Registered new tserver with Master: 080f175629c54c96b845358d2413c7f7 (127.2.172.67:40383)
I20260925 02:50:55.469905 9036 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:56239
I20260925 02:50:55.476435 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.172.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.125.keytab.
Entry for principal oryx/127.2.172.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.125.keytab.
I20260925 02:50:55.540383 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.125:38125
--webserver_interface=127.2.172.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.125.keytab
--principal=oryx/127.2.172.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:36101,127.2.172.125:38125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:50:55.881803 9363 heartbeater.cc:503] Master 127.2.172.126:36101 was elected leader, sending a full tablet report...
W20260925 02:50:55.968152 9504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:55.968613 9504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:55.968776 9504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:55.977557 9504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:55.977754 9504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:55.977871 9504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:55.977980 9504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:55.978063 9504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:55.990387 9504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:36101,127.2.172.125:38125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.125.keytab
--principal=oryx/127.2.172.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.125:38125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.172.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
W20260925 02:50:55.990988 9504 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.172.126:36101,127.2.172.125:38125'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 02:50:55.992579 9504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:55.994342 9504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:56.010038 9509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:56.010082 9512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:56.010039 9510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:56.010973 9504 server_base.cc:1034] running on GCE node
I20260925 02:50:56.012182 9504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:56.014328 9504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:56.015774 9504 hybrid_clock.cc:648] HybridClock initialized: now 1790304656015646 us; error 76 us; skew 500 ppm
I20260925 02:50:56.016247 9504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304656, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:56.020538 9504 init.cc:377] Logged in from keytab as oryx/127.2.172.125@KRBTEST.COM (short username oryx)
I20260925 02:50:56.022667 9504 webserver.cc:460] Webserver started at http://127.2.172.125:45947/ using document root <none> and password file <none>
I20260925 02:50:56.023705 9504 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:56.023854 9504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:56.024259 9504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:56.028272 9504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data/instance:
uuid: "3db79e82ab784ac3845a9e300581eea2"
format_stamp: "Formatted at 2026-09-25 02:50:56 on dist-test-slave-g1xh"
server_key: "6f8c564fea845de94d55138a5f88202f"
server_key_iv: "8ab0b665b3fffc19d6765250da0694ce"
server_key_version: "encryptionkey@0"
I20260925 02:50:56.029220 9504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/wal/instance:
uuid: "3db79e82ab784ac3845a9e300581eea2"
format_stamp: "Formatted at 2026-09-25 02:50:56 on dist-test-slave-g1xh"
server_key: "6f8c564fea845de94d55138a5f88202f"
server_key_iv: "8ab0b665b3fffc19d6765250da0694ce"
server_key_version: "encryptionkey@0"
I20260925 02:50:56.036060 9504 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 02:50:56.040557 9519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:56.042356 9504 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:50:56.042625 9504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/wal
uuid: "3db79e82ab784ac3845a9e300581eea2"
format_stamp: "Formatted at 2026-09-25 02:50:56 on dist-test-slave-g1xh"
server_key: "6f8c564fea845de94d55138a5f88202f"
server_key_iv: "8ab0b665b3fffc19d6765250da0694ce"
server_key_version: "encryptionkey@0"
I20260925 02:50:56.042928 9504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:56.080358 9504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:56.085539 9504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:56.085911 9504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:56.106271 9504 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.125:38125
I20260925 02:50:56.106315 9570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.125:38125 every 8 connection(s)
I20260925 02:50:56.107849 9504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
I20260925 02:50:56.110222 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 9504
I20260925 02:50:56.112766 9571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:56.122690 9571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.172.125@KRBTEST.COM
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304656, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.125@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:50:56.138064 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.112728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.2.172.125:38125)
0925 02:50:56.113667 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.113684 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.113739 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:50:56.116181 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.116212 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.116782 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.117283 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.118734 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.120930 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.122388 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.123485 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.123733 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.129688 (+ 5955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.129696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.129707 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.129742 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.133318 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.134234 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.134244 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.134254 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.134465 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.134880 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.134885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.134890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.135638 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.135882 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.136489 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.136971 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":450,"threads_started":1}
I20260925 02:50:56.143046 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.125725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39352)
0925 02:50:56.126821 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.126887 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.127076 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.127857 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.127864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.128819 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.129066 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.129075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.130457 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.130461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.132433 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.132448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.133230 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.133238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.133434 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.134074 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.134091 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.137034 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:56.140411 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.140425 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.140432 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.140932 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.141342 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.141345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.141347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.141477 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.142110 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:56.142116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:56.142334 (+ 218us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.142502 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.142751 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":360,"threads_started":1}
I20260925 02:50:56.143158 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.125777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.2.172.126:36101)
0925 02:50:56.127361 (+ 1584us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.127370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.127396 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:50:56.127484 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.127492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.127636 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.127794 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.129236 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.130234 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.132593 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.133024 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.133127 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.137240 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.137248 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.137256 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.137316 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.140264 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.141099 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.141104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.141106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.141237 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.141606 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.141610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.141612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.141983 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.142119 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.142618 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.142859 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1437,"thread_start_us":1293,"threads_started":1}
I20260925 02:50:56.144932 9571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304656, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.125@KRBTEST.COM for oryx/127.2.172.125@KRBTEST.COM
I20260925 02:50:56.158169 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.145573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:38125 (local address 127.0.0.1:43974)
0925 02:50:56.145774 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.145793 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.145881 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.146292 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.146296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.146653 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.146809 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.146818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.148081 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.148085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.148993 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.149005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.149755 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.149759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.149911 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.150507 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.150525 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.152743 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:56.155811 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.155815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.155817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.156172 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.156542 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.156545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.156547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.156670 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.157170 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:56.157173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:56.157330 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.157482 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.157780 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 02:50:56.158347 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.145681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.2.172.125:38125)
0925 02:50:56.145849 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.145853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.145869 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:50:56.146062 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.146066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.146120 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.146217 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.146976 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.147915 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.149134 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.149617 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.149732 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.152949 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.152953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.152957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.152985 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.155674 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.156290 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.156295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.156297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.156440 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.156777 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.156781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.156783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.157057 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.157176 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.157568 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.157937 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":73}
I20260925 02:50:56.163447 9571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: Bootstrap starting.
I20260925 02:50:56.166810 9571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:56.168027 9571 log.cc:824] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:56.170930 9571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: No bootstrap required, opened a new log
I20260925 02:50:56.178010 9571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } }
I20260925 02:50:56.178388 9571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:56.178488 9571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db79e82ab784ac3845a9e300581eea2, State: Initialized, Role: FOLLOWER
I20260925 02:50:56.179095 9571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } }
I20260925 02:50:56.180020 9578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } } }
I20260925 02:50:56.180423 9578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:50:56.181154 9571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:50:56.190807 9589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:50:56.190971 9589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:50:56.193701 9504 master_runner.cc:428] Detected that this master 3db79e82ab784ac3845a9e300581eea2 is joining an existing cluster
I20260925 02:50:56.193819 9504 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.172.125:38125
I20260925 02:50:56.203330 9578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:56.203684 9578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } }
I20260925 02:50:56.205433 9578 leader_election.cc:288] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0fda7481e654efa9d6911ae0af27ec6 (127.2.172.126:36101)
I20260925 02:50:56.206486 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3db79e82ab784ac3845a9e300581eea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fda7481e654efa9d6911ae0af27ec6" is_pre_election: true
I20260925 02:50:56.207880 9523 leader_election.cc:302] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 3db79e82ab784ac3845a9e300581eea2; no voters: e0fda7481e654efa9d6911ae0af27ec6
I20260925 02:50:56.208452 9578 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 02:50:56.215157 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.198344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.2.172.126:36101)
0925 02:50:56.198551 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.198558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.199323 (+ 765us) server_negotiation.cc:407] Connection header received
0925 02:50:56.199872 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.199881 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.200011 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.200106 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.202050 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.203169 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.205032 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.205611 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.205795 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.208814 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.208820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.208823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.208858 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.211832 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.212636 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.212643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.212647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.212822 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.213301 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.213308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.213313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.213713 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.213863 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.214489 (+ 626us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.214758 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:50:56.216754 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.198409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39364)
0925 02:50:56.199220 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.199244 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.199442 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.200422 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.200430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.201323 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.201820 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.201838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.203370 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.203378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.204857 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.204871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.205851 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.205857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.206052 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.206825 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.206992 (+ 167us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.208422 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:56.211972 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.211976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.211980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.212482 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.212978 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.212983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.212987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.213168 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.213951 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:56.213961 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:56.214168 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.215474 (+ 1306us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.215884 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":405,"threads_started":1}
I20260925 02:50:56.225891 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.199437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.2.172.125:38125)
0925 02:50:56.199630 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.199635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.206545 (+ 6910us) server_negotiation.cc:407] Connection header received
0925 02:50:56.206789 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.206794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.206875 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.206946 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.211972 (+ 5026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.213066 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.214947 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.215694 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.215801 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.218755 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.218761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.218765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.218798 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.222579 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.223472 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.223478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.223483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.223742 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.224187 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.224192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.224195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.224594 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.224753 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.225260 (+ 507us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.225495 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
WARNING: no policy specified for oryx/127.2.172.124@KRBTEST.COM; defaulting to no policy
I20260925 02:50:56.227106 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.199606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:38125 (local address 127.0.0.1:43976)
0925 02:50:56.206433 (+ 6827us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.206460 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.206641 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.207131 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.207135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.207957 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.211791 (+ 3834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.211809 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.213259 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.213276 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.214803 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.214817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.216285 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.216291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.216481 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.217430 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.217453 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.218569 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:56.222736 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.222744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.222750 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.223320 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.223884 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.223888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.223891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.224055 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.224843 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:56.224846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:56.224994 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.226305 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.226634 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6512,"thread_start_us":3614,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.124@KRBTEST.COM" created.
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304653, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.126@KRBTEST.COM for oryx/127.2.172.125@KRBTEST.COM
I20260925 02:50:56.245071 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.226784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.2.172.125:38125)
0925 02:50:56.227770 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.227776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.228310 (+ 534us) server_negotiation.cc:407] Connection header received
0925 02:50:56.228729 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.228734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.228826 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.228937 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.230414 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.231521 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.233581 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.234434 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.234583 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.238911 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.238916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.238920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.238958 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.241916 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.242739 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.242746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.242749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.242908 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.243331 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.243338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.243343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.243781 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.243943 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.244421 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.244667 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835}
I20260925 02:50:56.245122 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.226308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:38125 (local address 127.0.0.1:43978)
0925 02:50:56.228157 (+ 1849us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.228203 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.228491 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.229020 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.229029 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.229942 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.230253 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.230267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.231681 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.231687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.233373 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.233392 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.234641 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.234648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.234867 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.235739 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.235766 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.238662 (+ 2896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:56.242089 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.242106 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.242115 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.242603 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.243031 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.243037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.243040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.243207 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.243922 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:56.243931 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:56.244116 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.244327 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.244690 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1582,"thread_start_us":1421,"threads_started":1}
I20260925 02:50:56.247519 9036 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.172.125:38125: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3db79e82ab784ac3845a9e300581eea2"
member_type: NON_VOTER
last_known_addr {
host: "127.2.172.125"
port: 38125
}
attrs {
promote: true
}
}
dest_uuid: "e0fda7481e654efa9d6911ae0af27ec6"
cas_config_opid_index: -1
I20260925 02:50:56.249033 9036 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } }
I20260925 02:50:56.256735 9602 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3db79e82ab784ac3845a9e300581eea2 (127.2.172.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } } }
I20260925 02:50:56.257920 9546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:56.262342 9602 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.172.125:38125
I20260925 02:50:56.263265 9504 master.cc:537] Master@127.2.172.125:38125 shutting down...
I20260925 02:50:56.263198 9603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } } }
I20260925 02:50:56.263616 9603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:56.263520 9546 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 FOLLOWER]: Refusing update from remote peer e0fda7481e654efa9d6911ae0af27ec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 02:50:56.265318 9602 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.124.keytab.
Entry for principal oryx/127.2.172.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.124.keytab.
I20260925 02:50:56.268621 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.124:40083
--webserver_interface=127.2.172.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.124.keytab
--principal=oryx/127.2.172.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:36101,127.2.172.125:38125,127.2.172.124:40083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:56.278033 9023 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 02:50:56.279798 9504 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 02:50:56.280177 9023 consensus_peers.cc:487] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 -> Peer 3db79e82ab784ac3845a9e300581eea2 (127.2.172.125:38125): Couldn't send request to peer 3db79e82ab784ac3845a9e300581eea2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 02:50:56.280284 9504 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:56.280465 9504 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: stopping tablet replica
I20260925 02:50:56.287781 9504 master.cc:552] Master@127.2.172.125:38125 shutdown complete.
I20260925 02:50:56.287884 9504 master_runner.cc:305] Clearing existing system tablet
I20260925 02:50:56.288322 9504 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:56.288424 9504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:56.290786 9504 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:50:56.292315 9608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:56.292761 9504 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:56.293079 9504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:50:56.293172 9504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/wal
uuid: "3db79e82ab784ac3845a9e300581eea2"
format_stamp: "Formatted at 2026-09-25 02:50:56 on dist-test-slave-g1xh"
server_key: "6f8c564fea845de94d55138a5f88202f"
server_key_iv: "8ab0b665b3fffc19d6765250da0694ce"
server_key_version: "encryptionkey@0"
I20260925 02:50:56.297194 9504 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:56.301872 9504 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:50:56.302063 9504 log.cc:1196] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 02:50:56.302474 9504 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: Deleting consensus metadata
I20260925 02:50:56.302752 9504 master_runner.cc:315] Copying system tablet from 127.2.172.126:36101
I20260925 02:50:56.305804 9504 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.172.126:36101
I20260925 02:50:56.319761 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.306774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.2.172.126:36101)
0925 02:50:56.306996 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.307005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.307489 (+ 484us) server_negotiation.cc:407] Connection header received
0925 02:50:56.307732 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.307738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.307816 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.307929 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.308784 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.309604 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.310956 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.311798 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.311933 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.313857 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.313870 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.313874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.313908 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.316971 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.317600 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.317612 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.317616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.317742 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.318119 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.318126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.318129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.318411 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.318852 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.319101 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.319323 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:50:56.319784 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.306749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39366)
0925 02:50:56.307377 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.307393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.307558 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.307954 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.307958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.308390 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.308621 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.308630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.309754 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.309762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.310803 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.310815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.312178 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.312182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.312306 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.312931 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.312949 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.313640 (+ 691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:56.317100 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.317105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.317109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.317473 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.317864 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.317869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.317873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.318009 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.318615 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:56.318627 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:56.318817 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.319000 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.319326 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":397,"threads_started":1}
I20260925 02:50:56.321240 9056 tablet_copy_service.cc:140] P e0fda7481e654efa9d6911ae0af27ec6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3db79e82ab784ac3845a9e300581eea2 ({username='oryx', principal='oryx/127.2.172.125@KRBTEST.COM'} at 127.0.0.1:39366)
I20260925 02:50:56.321645 9056 tablet_copy_service.cc:161] P e0fda7481e654efa9d6911ae0af27ec6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3db79e82ab784ac3845a9e300581eea2 at {username='oryx', principal='oryx/127.2.172.125@KRBTEST.COM'} at 127.0.0.1:39366: session id = 3db79e82ab784ac3845a9e300581eea2-00000000000000000000000000000000
I20260925 02:50:56.326074 9056 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:50:56.328246 9504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:50:56.333113 9504 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: tablet copy: Starting download of 0 data blocks...
I20260925 02:50:56.333484 9504 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: tablet copy: Starting download of 1 WAL segments...
I20260925 02:50:56.336298 9504 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:50:56.339594 9056 tablet_copy_service.cc:342] P e0fda7481e654efa9d6911ae0af27ec6: Request end of tablet copy session 3db79e82ab784ac3845a9e300581eea2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.172.125@KRBTEST.COM'} at 127.0.0.1:39366
I20260925 02:50:56.339809 9056 tablet_copy_service.cc:433] P e0fda7481e654efa9d6911ae0af27ec6: ending tablet copy session 3db79e82ab784ac3845a9e300581eea2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3db79e82ab784ac3845a9e300581eea2
I20260925 02:50:56.350332 9504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:56.353276 9615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:56.359921 9616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:56.371927 9618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:56.377285 9504 server_base.cc:1034] running on GCE node
I20260925 02:50:56.377641 9504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:56.378311 9504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:56.379664 9504 hybrid_clock.cc:648] HybridClock initialized: now 1790304656379596 us; error 38 us; skew 500 ppm
I20260925 02:50:56.379897 9504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304656, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:56.381834 9504 init.cc:377] Logged in from keytab as oryx/127.2.172.125@KRBTEST.COM (short username oryx)
I20260925 02:50:56.387183 9504 webserver.cc:460] Webserver started at http://127.2.172.125:45447/ using document root <none> and password file <none>
I20260925 02:50:56.387526 9504 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:56.387676 9504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:56.390422 9504 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:50:56.392125 9625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:56.392556 9504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:50:56.392699 9504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/wal
uuid: "3db79e82ab784ac3845a9e300581eea2"
format_stamp: "Formatted at 2026-09-25 02:50:56 on dist-test-slave-g1xh"
server_key: "6f8c564fea845de94d55138a5f88202f"
server_key_iv: "8ab0b665b3fffc19d6765250da0694ce"
server_key_version: "encryptionkey@0"
I20260925 02:50:56.392887 9504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:56.408823 9504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:56.412575 9504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:56.456573 9504 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.125:38125
I20260925 02:50:56.456609 9676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.125:38125 every 8 connection(s)
I20260925 02:50:56.457700 9504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
I20260925 02:50:56.463706 9677 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:50:56.467633 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: Bootstrap starting.
I20260925 02:50:56.473701 9498 heartbeater.cc:503] Master 127.2.172.126:36101 was elected leader, sending a full tablet report...
I20260925 02:50:56.487078 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:50:56.487692 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: Bootstrap complete.
I20260925 02:50:56.488811 9677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } }
I20260925 02:50:56.489090 9677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3db79e82ab784ac3845a9e300581eea2, State: Initialized, Role: LEARNER
I20260925 02:50:56.489347 9677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } }
I20260925 02:50:56.490411 9680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } } }
I20260925 02:50:56.490664 9677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:56.490907 9680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:50:56.502983 9691 catalog_manager.cc:1259] Loaded cluster ID: d9373e52acd046568f4e4682f8104893
I20260925 02:50:56.503284 9691 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: loading cluster ID for follower catalog manager: success
I20260925 02:50:56.507138 9691 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: acquiring CA information for follower catalog manager: success
I20260925 02:50:56.510202 9691 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 02:50:56.708918 9605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:56.709285 9605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:56.709447 9605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:56.717438 9605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:56.717589 9605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:56.717695 9605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:56.717793 9605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:56.717883 9605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:56.725595 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.718425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.2.172.125:38125)
0925 02:50:56.719042 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.719046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.719064 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:50:56.719127 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.719131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.719204 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.719315 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:56.719876 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.720999 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.723462 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.724498 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.724610 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.724812 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.725229 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.725337 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":354,"threads_started":1}
I20260925 02:50:56.725621 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.718217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:38125 (local address 127.0.0.1:43992)
0925 02:50:56.718427 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.718449 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.718580 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.719402 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.719407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.719440 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:56.719691 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.719704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.721174 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.721180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.723241 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.723260 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.724690 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.724696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.724998 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.725073 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.725181 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.725334 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 02:50:56.726982 9652 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 02:50:56.727653 9680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0fda7481e654efa9d6911ae0af27ec6. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } } }
I20260925 02:50:56.727957 9680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:50:56.731120 9605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:36101,127.2.172.125:38125,127.2.172.124:40083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.124.keytab
--principal=oryx/127.2.172.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.124:40083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.172.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:56.732609 9605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:56.733960 9605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:56.746338 9700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:56.746338 9698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:56.746410 9697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:56.747143 9605 server_base.cc:1034] running on GCE node
I20260925 02:50:56.748026 9605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:56.749544 9605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:56.750780 9605 hybrid_clock.cc:648] HybridClock initialized: now 1790304656750738 us; error 41 us; skew 500 ppm
I20260925 02:50:56.751130 9605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304656, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:56.754467 9605 init.cc:377] Logged in from keytab as oryx/127.2.172.124@KRBTEST.COM (short username oryx)
I20260925 02:50:56.756115 9605 webserver.cc:460] Webserver started at http://127.2.172.124:33259/ using document root <none> and password file <none>
I20260925 02:50:56.756824 9605 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:56.756974 9605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:56.757287 9605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:56.760634 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data/instance:
uuid: "e197a1b4dfb94d7fb94f67e590716711"
format_stamp: "Formatted at 2026-09-25 02:50:56 on dist-test-slave-g1xh"
server_key: "aeff7e6c68e3c418225d1932daaeb057"
server_key_iv: "39a444f6a963f3178e5280a765eee8af"
server_key_version: "encryptionkey@0"
I20260925 02:50:56.761602 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/wal/instance:
uuid: "e197a1b4dfb94d7fb94f67e590716711"
format_stamp: "Formatted at 2026-09-25 02:50:56 on dist-test-slave-g1xh"
server_key: "aeff7e6c68e3c418225d1932daaeb057"
server_key_iv: "39a444f6a963f3178e5280a765eee8af"
server_key_version: "encryptionkey@0"
I20260925 02:50:56.767119 9605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:50:56.770962 9707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:56.772505 9605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:56.772713 9605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/wal
uuid: "e197a1b4dfb94d7fb94f67e590716711"
format_stamp: "Formatted at 2026-09-25 02:50:56 on dist-test-slave-g1xh"
server_key: "aeff7e6c68e3c418225d1932daaeb057"
server_key_iv: "39a444f6a963f3178e5280a765eee8af"
server_key_version: "encryptionkey@0"
I20260925 02:50:56.772931 9605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:56.796564 9605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:56.800853 9605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:56.801151 9605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:56.822517 9605 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.124:40083
I20260925 02:50:56.822527 9759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.124:40083 every 8 connection(s)
I20260925 02:50:56.823984 9605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
I20260925 02:50:56.825619 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 9605
I20260925 02:50:56.828815 9760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:56.837085 9760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.172.124@KRBTEST.COM
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304656, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.124@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:50:56.851399 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.827889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.2.172.124:40083)
0925 02:50:56.828811 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.828827 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.829079 (+ 252us) server_negotiation.cc:407] Connection header received
0925 02:50:56.830844 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.830864 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.831215 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.831542 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.833346 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.834736 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.836557 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.837223 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.837360 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.842654 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.842664 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.842678 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.842733 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.846393 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.847229 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.847235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.847241 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.847478 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.847917 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.847925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.847931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.848686 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.849012 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.849557 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.850009 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":388,"threads_started":1}
I20260925 02:50:56.856745 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.840340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.2.172.126:36101)
0925 02:50:56.841719 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.841727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.841747 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:50:56.841807 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.841812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.841896 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.842033 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.843477 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.844301 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.846535 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.847185 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.847353 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.851230 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.851237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.851242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.851285 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.854200 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.854837 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.854841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.854843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.854969 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.855366 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.855370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.855372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.855771 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.855897 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.856291 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.856467 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259,"thread_start_us":1155,"threads_started":1}
I20260925 02:50:56.856743 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.840150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39376)
0925 02:50:56.841042 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.841084 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.841253 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.842099 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.842104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.842938 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.843298 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.843313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.844479 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.844484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.846342 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.846356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.847459 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.847466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.847685 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.848228 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.848254 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.851028 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:56.854327 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.854335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.854340 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.854728 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.855065 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.855069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.855071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.855195 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.855890 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:56.855896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:56.856039 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.856194 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.856468 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":427,"threads_started":1}
I20260925 02:50:56.858072 9760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304656, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.124@KRBTEST.COM for oryx/127.2.172.125@KRBTEST.COM
I20260925 02:50:56.871685 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.858661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:38125 (local address 127.0.0.1:44004)
0925 02:50:56.858886 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.858902 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.859026 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.859402 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.859406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.859802 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.859931 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.859939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.860980 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.860986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.862012 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.862023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.862748 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.862754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.862934 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.863506 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.863522 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.865683 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:56.868969 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.868973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.868976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.869354 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.869813 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.869816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.869819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.869961 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.870630 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:56.870635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:56.870795 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.870943 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.871256 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 02:50:56.871683 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.858906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.2.172.125:38125)
0925 02:50:56.859122 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.859129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.859153 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:50:56.859198 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.859202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.859266 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.859375 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.860071 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.860843 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.862162 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.862575 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.862709 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.865886 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.865891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.865895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.865928 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.868843 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.869473 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.869478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.869480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.869683 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.870066 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.870069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.870071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.870486 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.870652 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.871032 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.871256 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:50:56.872805 9760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304656, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.124@KRBTEST.COM for oryx/127.2.172.124@KRBTEST.COM
I20260925 02:50:56.885747 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.873541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:40083 (local address 127.0.0.1:52768)
0925 02:50:56.873777 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.873796 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.873912 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.874194 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.874198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.874597 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.874740 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.874748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.875765 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.875773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.876648 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.876659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.877437 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.877444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.877746 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.878315 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.878332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.880346 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:56.883319 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.883322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.883325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.883790 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.884139 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.884142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.884144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.884305 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.884814 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:56.884818 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:56.884983 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.885139 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.885384 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 02:50:56.885985 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.873615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.2.172.124:40083)
0925 02:50:56.873777 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.873785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.873864 (+ 79us) server_negotiation.cc:407] Connection header received
0925 02:50:56.874026 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.874031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.874088 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.874180 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.874867 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.875644 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.876786 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.877258 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.877402 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.880518 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.880523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.880526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.880554 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.883165 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.883896 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.883900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.883902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.884030 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.884414 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.884422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.884425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.884681 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.884843 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.885260 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.885567 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":50}
I20260925 02:50:56.890794 9760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: Bootstrap starting.
I20260925 02:50:56.894035 9760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:56.895038 9760 log.cc:824] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:56.897746 9760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: No bootstrap required, opened a new log
I20260925 02:50:56.905342 9760 raft_consensus.cc:348] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } }
I20260925 02:50:56.905750 9760 raft_consensus.cc:374] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:56.905889 9760 raft_consensus.cc:729] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e197a1b4dfb94d7fb94f67e590716711, State: Initialized, Role: FOLLOWER
I20260925 02:50:56.906416 9760 consensus_queue.cc:260] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } }
I20260925 02:50:56.907256 9767 sys_catalog.cc:455] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } } }
I20260925 02:50:56.907626 9767 sys_catalog.cc:458] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:50:56.908502 9760 sys_catalog.cc:564] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:50:56.919448 9778 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:50:56.919648 9778 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:50:56.924566 9605 master_runner.cc:428] Detected that this master e197a1b4dfb94d7fb94f67e590716711 is joining an existing cluster
I20260925 02:50:56.924724 9605 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.172.124:40083
I20260925 02:50:56.947113 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.928897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39388)
0925 02:50:56.929598 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.929620 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.929833 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.930407 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.930411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.931143 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.931527 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.931540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.933499 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.933503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.935212 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.935226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.936517 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.936524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.936717 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.937470 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.937496 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.939211 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:56.943791 (+ 4580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.943796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.943800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.944325 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.944799 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.944804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.944807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.945000 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.945771 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:56.945776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:56.946024 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.946243 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.946588 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":1,"thread_start_us":391,"threads_started":1}
I20260925 02:50:56.948048 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.928809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.2.172.126:36101)
0925 02:50:56.928999 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.929007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.929705 (+ 698us) server_negotiation.cc:407] Connection header received
0925 02:50:56.930000 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.930008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.930112 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.930187 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.931746 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.932946 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.935387 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.936115 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.936215 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.939465 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.939474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.939479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.939523 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.943610 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.944487 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.944493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.944497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.944651 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.945144 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.945149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.945152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.945535 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.945707 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.947443 (+ 1736us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.947715 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:50:56.956645 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.928992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:40083 (local address 127.0.0.1:52770)
0925 02:50:56.940124 (+ 11132us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.940182 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.940366 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.940844 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.940848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.942003 (+ 1155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.942530 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.942542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.944179 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.944184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.945867 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.945880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.947107 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.947114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.947294 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.947930 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.947957 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.949081 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:56.953195 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.953200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.953204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.953837 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.954303 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.954310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.954313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.954435 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.954973 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:56.954978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:56.955165 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.955339 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.955787 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10968,"spinlock_wait_cycles":22272,"thread_start_us":10769,"threads_started":1}
I20260925 02:50:56.957587 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 9097
I20260925 02:50:56.957691 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.928863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.2.172.124:40083)
0925 02:50:56.929076 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.929081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.940261 (+ 11180us) server_negotiation.cc:407] Connection header received
0925 02:50:56.940513 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.940519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.940590 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.940670 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.942702 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.943988 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.946016 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.946724 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.946813 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.949283 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.949289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.949294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.949327 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.953064 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.953993 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.954001 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.954008 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.954167 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.954548 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.954553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.954556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.954860 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.955051 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.955435 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.955922 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":276,"server-negotiator.queue_time_us":64}
I20260925 02:50:56.959829 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.930038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.2.172.125:38125)
0925 02:50:56.930271 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.930279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.942085 (+ 11806us) server_negotiation.cc:407] Connection header received
0925 02:50:56.942362 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.942371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.942463 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.942541 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.944168 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.945257 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.947650 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.948353 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.948453 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.952183 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.952191 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.952196 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.952239 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.955921 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.956857 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.956862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.956866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.957098 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.957665 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.957681 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.957684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.958185 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.958324 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.959052 (+ 728us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.959331 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:50:56.960955 9786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.930429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:38125 (local address 127.0.0.1:44006)
0925 02:50:56.941980 (+ 11551us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.941999 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.942203 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.942852 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.942857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.943664 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.943997 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.944005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.945434 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.945439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.947143 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.947153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.949295 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.949300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.949496 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.950590 (+ 1094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.950615 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.951952 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:56.956091 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.956096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.956099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.956704 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.957262 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:56.957266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:56.957270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:56.957525 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:56.958403 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:56.958408 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:56.958741 (+ 333us) client_negotiation.cc:769] Sending connection context
0925 02:50:56.960072 (+ 1331us) client_negotiation.cc:240] Negotiation successful
0925 02:50:56.960423 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11371,"thread_start_us":11023,"threads_started":1}
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.172.65@KRBTEST.COM
I20260925 02:50:56.980039 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.959656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.2.172.65:39097)
0925 02:50:56.960701 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.960709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.960744 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:50:56.960812 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.960819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.961135 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.961317 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.963018 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.964266 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.966234 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.966834 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.966962 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.972876 (+ 5914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.972893 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.972908 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.972947 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:56.976653 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.977421 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.977429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.977432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.977659 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:56.978088 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:56.978092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:56.978095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:56.978517 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:56.978860 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:56.979197 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 02:50:56.979481 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":450,"threads_started":1}
I20260925 02:50:56.981464 9222 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 25 02:50:56 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304653, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.126@KRBTEST.COM for oryx/127.2.172.124@KRBTEST.COM
I20260925 02:50:57.005131 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.985581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.2.172.124:40083)
0925 02:50:56.985789 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:56.985798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:56.985823 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:50:56.985944 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:56.985952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:56.986114 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:56.986256 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:56.987798 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.989326 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.991773 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.992994 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.993189 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.998796 (+ 5607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:56.998801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:56.998805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:56.998840 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:57.002237 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:57.002915 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:57.002926 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:57.002928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:57.003039 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:57.003494 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:57.003501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:57.003507 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:57.003934 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:57.004092 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:57.004474 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 02:50:57.004705 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:50:57.005857 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:56.984607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:40083 (local address 127.0.0.1:52772)
0925 02:50:56.984936 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:56.984958 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:56.985093 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:56.986534 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:56.986542 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:56.987197 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:56.987478 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.987496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.989534 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.989543 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.991475 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:56.991497 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:56.993580 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:56.993589 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:56.993890 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:56.994843 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:56.994870 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:56.998545 (+ 3675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:57.002386 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:57.002391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:57.002395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:57.002783 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:57.003164 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:57.003169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:57.003172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:57.003353 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:57.004039 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:57.004043 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:57.004207 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 02:50:57.004694 (+ 487us) client_negotiation.cc:240] Negotiation successful
0925 02:50:57.005110 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260925 02:50:57.007450 9036 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.172.124:40083: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711"
member_type: NON_VOTER
last_known_addr {
host: "127.2.172.124"
port: 40083
}
attrs {
promote: true
}
}
dest_uuid: "e0fda7481e654efa9d6911ae0af27ec6"
cas_config_opid_index: 5
I20260925 02:50:57.008452 9036 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } }
I20260925 02:50:57.010304 9603 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e197a1b4dfb94d7fb94f67e590716711 (127.2.172.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.011080 9652 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 LEARNER]: Refusing update from remote peer e0fda7481e654efa9d6911ae0af27ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 02:50:57.012678 9734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:57.014611 9603 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.172.124:40083
I20260925 02:50:57.015424 9605 master.cc:537] Master@127.2.172.124:40083 shutting down...
I20260925 02:50:57.017333 9734 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 1 FOLLOWER]: Refusing update from remote peer e0fda7481e654efa9d6911ae0af27ec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 02:50:57.022534 9790 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 02:50:57.030350 9605 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:57.030819 9605 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:57.031075 9605 tablet_replica.cc:330] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: stopping tablet replica
I20260925 02:50:57.034973 9791 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:50:57.038262 9022 connection.cc:535] client connection to 127.2.172.124:40083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:50:57.040126 9652 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e197a1b4dfb94d7fb94f67e590716711 (127.2.172.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.045787 9680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.046147 9680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:50:57.050747 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.050243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:40083 (local address 127.0.0.1:52788)
0925 02:50:57.050421 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:57.050558 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:40083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 02:50:57.051115 9605 master.cc:552] Master@127.2.172.124:40083 shutdown complete.
I20260925 02:50:57.051259 9605 master_runner.cc:305] Clearing existing system tablet
W20260925 02:50:57.051187 9022 consensus_peers.cc:487] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 -> Peer e197a1b4dfb94d7fb94f67e590716711 (127.2.172.124:40083): Couldn't send request to peer e197a1b4dfb94d7fb94f67e590716711. Status: Network error: Client connection negotiation failed: client connection to 127.2.172.124:40083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:50:57.051867 9605 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:57.052343 9791 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6: attempting to promote NON_VOTER 3db79e82ab784ac3845a9e300581eea2 to VOTER
I20260925 02:50:57.052403 9790 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.052755 9790 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:57.052073 9605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:57.053356 9791 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } }
I20260925 02:50:57.054662 9603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: NON_VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: true } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.054997 9603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:57.055757 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.055062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:40083 (local address 127.0.0.1:52800)
0925 02:50:57.055301 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:57.055417 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:40083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 02:50:57.056157 9652 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 LEARNER]: Refusing update from remote peer e0fda7481e654efa9d6911ae0af27ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 02:50:57.056761 9603 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 02:50:57.057034 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.056624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:40083 (local address 127.0.0.1:52814)
0925 02:50:57.056824 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:57.056904 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:40083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260925 02:50:57.057308 9022 consensus_peers.cc:487] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 -> Peer e197a1b4dfb94d7fb94f67e590716711 (127.2.172.124:40083): Couldn't send request to peer e197a1b4dfb94d7fb94f67e590716711. Status: Network error: Client connection negotiation failed: client connection to 127.2.172.124:40083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:50:57.059175 9789 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3db79e82ab784ac3845a9e300581eea2 (127.2.172.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.059868 9651 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3db79e82ab784ac3845a9e300581eea2 (127.2.172.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.063408 9791 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.063714 9791 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:57.063517 9603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.063875 9603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:57.064010 9605 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260925 02:50:57.064599 9680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.064851 9680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:50:57.065919 9796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:57.066237 9605 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:57.066527 9605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:50:57.066640 9605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/wal
uuid: "e197a1b4dfb94d7fb94f67e590716711"
format_stamp: "Formatted at 2026-09-25 02:50:56 on dist-test-slave-g1xh"
server_key: "aeff7e6c68e3c418225d1932daaeb057"
server_key_iv: "39a444f6a963f3178e5280a765eee8af"
server_key_version: "encryptionkey@0"
I20260925 02:50:57.070292 9605 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:57.075173 9605 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:50:57.075369 9605 log.cc:1196] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 02:50:57.075860 9605 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: Deleting consensus metadata
I20260925 02:50:57.076108 9605 master_runner.cc:315] Copying system tablet from 127.2.172.126:36101
I20260925 02:50:57.078614 9605 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: tablet copy: Beginning tablet copy session from remote peer at address 127.2.172.126:36101
I20260925 02:50:57.096711 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.079434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39392 (local address 127.2.172.126:36101)
0925 02:50:57.079728 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:57.079738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:57.084128 (+ 4390us) server_negotiation.cc:407] Connection header received
0925 02:50:57.084389 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:57.084394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:57.084499 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:57.084619 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:57.085487 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.086383 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.087960 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.088418 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.088573 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:57.090424 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:57.090430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:57.090435 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:57.090477 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:57.093805 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:57.094660 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:57.094669 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:57.094673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:57.094825 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:57.095213 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:57.095219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:57.095222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:57.095503 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:57.095697 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:57.096108 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 02:50:57.096323 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260925 02:50:57.096817 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.079399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39392)
0925 02:50:57.083989 (+ 4590us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:57.084009 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:57.084233 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:57.084652 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:57.084656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:57.085089 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:57.085335 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:57.085345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.086522 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.086528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:57.087803 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:57.087816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.088577 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.088583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:57.088780 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:57.089465 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:57.089484 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:57.090207 (+ 723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:57.093960 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:57.093966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:57.093970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:57.094543 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:57.094940 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:57.094945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:57.094949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:57.095100 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:57.095660 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:57.095665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:57.095843 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 02:50:57.095989 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 02:50:57.096335 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4454,"thread_start_us":4351,"threads_started":1}
I20260925 02:50:57.097743 9056 tablet_copy_service.cc:140] P e0fda7481e654efa9d6911ae0af27ec6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e197a1b4dfb94d7fb94f67e590716711 ({username='oryx', principal='oryx/127.2.172.124@KRBTEST.COM'} at 127.0.0.1:39392)
I20260925 02:50:57.097923 9056 tablet_copy_service.cc:161] P e0fda7481e654efa9d6911ae0af27ec6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e197a1b4dfb94d7fb94f67e590716711 at {username='oryx', principal='oryx/127.2.172.124@KRBTEST.COM'} at 127.0.0.1:39392: session id = e197a1b4dfb94d7fb94f67e590716711-00000000000000000000000000000000
I20260925 02:50:57.100474 9056 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:50:57.102082 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:50:57.107689 9605 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: tablet copy: Starting download of 0 data blocks...
I20260925 02:50:57.107993 9605 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: tablet copy: Starting download of 1 WAL segments...
I20260925 02:50:57.110423 9605 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:50:57.114128 9056 tablet_copy_service.cc:342] P e0fda7481e654efa9d6911ae0af27ec6: Request end of tablet copy session e197a1b4dfb94d7fb94f67e590716711-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.172.124@KRBTEST.COM'} at 127.0.0.1:39392
I20260925 02:50:57.114280 9056 tablet_copy_service.cc:433] P e0fda7481e654efa9d6911ae0af27ec6: ending tablet copy session e197a1b4dfb94d7fb94f67e590716711-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e197a1b4dfb94d7fb94f67e590716711
I20260925 02:50:57.124055 9605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:57.126938 9803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:57.133409 9804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:57.141851 9605 server_base.cc:1034] running on GCE node
W20260925 02:50:57.142161 9806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:57.142671 9605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:57.143611 9605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:57.151162 9605 hybrid_clock.cc:648] HybridClock initialized: now 1790304657149020 us; error 2032 us; skew 500 ppm
I20260925 02:50:57.151648 9605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:57 dist-test-slave-g1xh krb5kdc[8991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304657, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:57.154065 9605 init.cc:377] Logged in from keytab as oryx/127.2.172.124@KRBTEST.COM (short username oryx)
I20260925 02:50:57.157287 9605 webserver.cc:460] Webserver started at http://127.2.172.124:40279/ using document root <none> and password file <none>
I20260925 02:50:57.157424 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 9097
I20260925 02:50:57.159811 9605 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:57.160054 9605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:57.163826 9605 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 02:50:57.166131 9813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:57.166627 9605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:50:57.166761 9605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/wal
uuid: "e197a1b4dfb94d7fb94f67e590716711"
format_stamp: "Formatted at 2026-09-25 02:50:56 on dist-test-slave-g1xh"
server_key: "aeff7e6c68e3c418225d1932daaeb057"
server_key_iv: "39a444f6a963f3178e5280a765eee8af"
server_key_version: "encryptionkey@0"
I20260925 02:50:57.166939 9605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:57.177944 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:39097
--local_ip_for_outbound_sockets=127.2.172.65
--tserver_master_addrs=127.2.172.126:36101
--webserver_port=43923
--webserver_interface=127.2.172.65
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab
--principal=oryx/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.172.126:36101,127.2.172.125:38125
--tserver_master_addrs=127.2.172.126:36101,127.2.172.125:38125,127.2.172.124:40083 with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:50:57.195448 9605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:57.199898 9605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:57.235659 9605 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.124:40083
I20260925 02:50:57.235873 9865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.124:40083 every 8 connection(s)
I20260925 02:50:57.236773 9605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
I20260925 02:50:57.242306 9866 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:50:57.246259 9866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: Bootstrap starting.
I20260925 02:50:57.265982 9866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:50:57.266548 9866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: Bootstrap complete.
I20260925 02:50:57.267817 9866 raft_consensus.cc:348] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } }
I20260925 02:50:57.268168 9866 raft_consensus.cc:729] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e197a1b4dfb94d7fb94f67e590716711, State: Initialized, Role: LEARNER
I20260925 02:50:57.268481 9866 consensus_queue.cc:260] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } }
I20260925 02:50:57.269366 9869 sys_catalog.cc:455] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.269670 9866 sys_catalog.cc:564] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:57.269850 9869 sys_catalog.cc:458] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:50:57.283113 9880 catalog_manager.cc:1259] Loaded cluster ID: d9373e52acd046568f4e4682f8104893
I20260925 02:50:57.283241 9880 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: loading cluster ID for follower catalog manager: success
I20260925 02:50:57.287034 9880 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: acquiring CA information for follower catalog manager: success
I20260925 02:50:57.289634 9880 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:50:57.453976 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.446377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:40083 (local address 127.0.0.1:52826)
0925 02:50:57.446577 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:57.446600 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:57.446767 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:57.447720 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:57.447726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:57.447766 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:57.448033 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:57.448049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.449353 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.449358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:57.451767 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:57.451788 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.453180 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.453185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:57.453371 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:57.453407 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:50:57.453502 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 02:50:57.453680 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 02:50:57.453991 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.446570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.2.172.124:40083)
0925 02:50:57.447331 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:57.447336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:57.447355 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:50:57.447414 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:57.447417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:57.447489 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:57.447668 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:57.448203 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.449186 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.451950 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.452992 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.453145 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:57.453464 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:57.453521 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 02:50:57.453657 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":501,"threads_started":1}
I20260925 02:50:57.455240 9841 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 02:50:57.455849 9869 sys_catalog.cc:455] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0fda7481e654efa9d6911ae0af27ec6. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.456142 9869 sys_catalog.cc:458] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: This master's current role is: LEARNER
W20260925 02:50:57.578423 9818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:57.578850 9818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:57.579021 9818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:57.587600 9818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:57.587764 9818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:57.587946 9818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:50:57.599009 9818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab
--principal=oryx/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:39097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=43923
--tserver_master_addrs=127.2.172.126:36101,127.2.172.125:38125,127.2.172.124:40083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:57.600584 9818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:57.601927 9818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:57.616015 9886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:57.616348 9887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:57.616647 9889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:57.621646 9818 server_base.cc:1034] running on GCE node
I20260925 02:50:57.622406 9818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:57.623502 9818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:57.624805 9818 hybrid_clock.cc:648] HybridClock initialized: now 1790304657624729 us; error 64 us; skew 500 ppm
I20260925 02:50:57.625149 9818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:57 dist-test-slave-g1xh krb5kdc[8991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304657, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:57.628545 9818 init.cc:377] Logged in from keytab as oryx/127.2.172.65@KRBTEST.COM (short username oryx)
I20260925 02:50:57.630074 9818 webserver.cc:460] Webserver started at http://127.2.172.65:43923/ using document root <none> and password file <none>
I20260925 02:50:57.630896 9818 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:57.631062 9818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:57.636812 9818 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:50:57.639909 9896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:57.641566 9818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:50:57.641716 9818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "30ab128117884673903a3c39d748b7a2"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-g1xh"
server_key: "ee2aae5ab861084a57279c700e314ae2"
server_key_iv: "ea7910d83e25fa867b05d5072896ebfb"
server_key_version: "encryptionkey@0"
I20260925 02:50:57.642709 9818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:57.672338 9818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:57.677279 9818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:57.677587 9818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:57.678947 9818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:57.680594 9818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:57.680742 9818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:57.680871 9818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:57.680959 9818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:57.718173 9818 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:39097
I20260925 02:50:57.718271 10010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:39097 every 8 connection(s)
I20260925 02:50:57.719746 9818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:57.723114 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 9818
I20260925 02:50:57.723472 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 9233
Sep 25 02:50:57 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304657, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.125@KRBTEST.COM
Sep 25 02:50:57 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304657, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
Sep 25 02:50:57 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304657, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.124@KRBTEST.COM
Sep 25 02:50:57 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.172.66@KRBTEST.COM
I20260925 02:50:57.766517 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.725563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:39885 (local address 127.2.172.124:40083)
0925 02:50:57.725760 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:57.725768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:57.741670 (+ 15902us) server_negotiation.cc:407] Connection header received
0925 02:50:57.741812 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:57.741821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:57.741931 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:57.742060 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:57.745430 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.746625 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.750715 (+ 4090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.751412 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.751522 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:57.759094 (+ 7572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:57.759103 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:57.759109 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:57.759157 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:57.762838 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:57.763792 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:57.763799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:57.763803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:57.763954 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:57.764809 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:57.764815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:57.764820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:57.765194 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:57.765322 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:57.765898 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 02:50:57.766151 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:50:57.767724 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.724867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.2.172.66:33489)
0925 02:50:57.740104 (+ 15237us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:57.740117 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:57.744337 (+ 4220us) server_negotiation.cc:407] Connection header received
0925 02:50:57.744541 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:57.744550 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:57.744849 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:57.745017 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:57.747263 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.748825 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.750778 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.751708 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.751826 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:57.759581 (+ 7755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:57.759603 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:57.759621 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:57.759681 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:57.763719 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:57.764610 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:57.764620 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:57.764624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:57.764791 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:57.765260 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:57.765265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:57.765268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:57.765670 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:57.765945 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:57.767015 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0925 02:50:57.767281 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14994,"thread_start_us":16283,"threads_started":1}
I20260925 02:50:57.768052 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.725819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:53631 (local address 127.2.172.125:38125)
0925 02:50:57.743077 (+ 17258us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:57.743087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:57.743122 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:50:57.743198 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:57.743206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:57.743325 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:57.743477 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:57.745437 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.746715 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.750745 (+ 4030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.751610 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.751736 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:57.757585 (+ 5849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:57.757596 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:57.757603 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:57.757651 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:57.762198 (+ 4547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:57.763318 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:57.763324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:57.763329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:57.763506 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:57.764152 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:57.764157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:57.764161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:57.764556 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:57.764705 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:57.767328 (+ 2623us) server_negotiation.cc:299] Negotiation successful
0925 02:50:57.767651 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17080,"thread_start_us":16946,"threads_started":1}
I20260925 02:50:57.768285 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.724492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:40083 (local address 127.2.172.65:39885)
0925 02:50:57.741517 (+ 17025us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:57.741539 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:57.741679 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:57.742319 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:57.742327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:57.743937 (+ 1610us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:57.745175 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:57.745221 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.746923 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.746928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:57.750489 (+ 3561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:57.750514 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.751976 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.751986 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:57.752542 (+ 556us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:57.753379 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:57.753421 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:57.758839 (+ 5418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:57.763010 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:57.763016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:57.763020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:57.763637 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:57.764111 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:57.764118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:57.764122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:57.764667 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:57.765402 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:57.765407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:57.765568 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 02:50:57.766863 (+ 1295us) client_negotiation.cc:240] Negotiation successful
0925 02:50:57.767150 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14993,"mutex_wait_us":731,"spinlock_wait_cycles":787584,"thread_start_us":14784,"threads_started":1}
I20260925 02:50:57.768908 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.724989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:53819 (local address 127.2.172.126:36101)
0925 02:50:57.740587 (+ 15598us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:57.740596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:57.740623 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:50:57.740704 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:57.740709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:57.740801 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:57.740942 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:57.745438 (+ 4496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.746642 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.750741 (+ 4099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.751522 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.751713 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:57.758097 (+ 6384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:57.758105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:57.758111 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:57.758160 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:57.763541 (+ 5381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:57.764785 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:57.764791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:57.764795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:57.764964 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:57.765474 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:57.765481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:57.765485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:57.765857 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:57.765990 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:57.767370 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0925 02:50:57.767651 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15445,"thread_start_us":15320,"threads_started":1}
I20260925 02:50:57.768393 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.723529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.65:53819)
0925 02:50:57.730853 (+ 7324us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:57.730933 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:57.738519 (+ 7586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:57.742235 (+ 3716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:57.742255 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:57.744733 (+ 2478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:57.745182 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:57.745221 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.746860 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.746866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:57.750486 (+ 3620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:57.750513 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.752064 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.752070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:57.752532 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:57.753689 (+ 1157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:57.753710 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:57.757778 (+ 4068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:57.763725 (+ 5947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:57.763731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:57.763737 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:57.764645 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:57.765105 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:57.765110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:57.765114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:57.765309 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:57.766063 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:57.766068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:57.766245 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 02:50:57.766388 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 02:50:57.766942 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":621,"spinlock_wait_cycles":49408,"thread_start_us":599,"threads_started":1}
I20260925 02:50:57.768618 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:57.725391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:38125 (local address 127.2.172.65:53631)
0925 02:50:57.741036 (+ 15645us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:57.741065 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:57.741245 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:57.743532 (+ 2287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:57.743537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:57.744139 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:57.745177 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:57.745221 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.746879 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.746883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:57.750485 (+ 3602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:57.750513 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:57.752056 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:57.752063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:57.752518 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:57.753649 (+ 1131us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:57.753677 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:57.757203 (+ 3526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:57.762392 (+ 5189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:57.762405 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:57.762428 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:57.763135 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:57.763685 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:57.763694 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:57.763698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:57.763996 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:57.764793 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:57.764802 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:57.765319 (+ 517us) client_negotiation.cc:769] Sending connection context
0925 02:50:57.765699 (+ 380us) client_negotiation.cc:240] Negotiation successful
0925 02:50:57.766300 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15464,"mutex_wait_us":524,"spinlock_wait_cycles":23552,"thread_start_us":15289,"threads_started":1}
I20260925 02:50:57.770038 9358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:57.770160 10011 heartbeater.cc:347] Connected to a master server at 127.2.172.124:40083
I20260925 02:50:57.770525 10011 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:57.771194 10013 heartbeater.cc:347] Connected to a master server at 127.2.172.125:38125
I20260925 02:50:57.771370 10013 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:57.771697 10011 heartbeater.cc:511] Master 127.2.172.124:40083 requested a full tablet report, sending...
I20260925 02:50:57.772027 10013 heartbeater.cc:511] Master 127.2.172.125:38125 requested a full tablet report, sending...
I20260925 02:50:57.772169 10012 heartbeater.cc:347] Connected to a master server at 127.2.172.126:36101
I20260925 02:50:57.772302 10012 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:57.772904 10012 heartbeater.cc:511] Master 127.2.172.126:36101 requested a full tablet report, sending...
I20260925 02:50:57.774114 9036 ts_manager.cc:194] Re-registered known tserver with Master: 30ab128117884673903a3c39d748b7a2 (127.2.172.65:39097)
I20260925 02:50:57.774134 9642 ts_manager.cc:194] Registered new tserver with Master: 30ab128117884673903a3c39d748b7a2 (127.2.172.65:39097)
I20260925 02:50:57.774353 9831 ts_manager.cc:194] Registered new tserver with Master: 30ab128117884673903a3c39d748b7a2 (127.2.172.65:39097)
I20260925 02:50:57.775306 9036 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:53819
I20260925 02:50:57.834210 9603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: NON_VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: true } } }
I20260925 02:50:57.834709 9603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:57.836089 9603 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6: attempting to promote NON_VOTER e197a1b4dfb94d7fb94f67e590716711 to VOTER
I20260925 02:50:57.836712 9603 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } }
I20260925 02:50:57.838665 9841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 1 LEARNER]: Refusing update from remote peer e0fda7481e654efa9d6911ae0af27ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 02:50:57.838639 9651 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 FOLLOWER]: Refusing update from remote peer e0fda7481e654efa9d6911ae0af27ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 02:50:57.839232 10024 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 02:50:57.839745 10023 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 02:50:57.842624 10024 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e197a1b4dfb94d7fb94f67e590716711 (127.2.172.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } } }
I20260925 02:50:57.843082 9651 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e197a1b4dfb94d7fb94f67e590716711 (127.2.172.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } } }
I20260925 02:50:57.846747 9603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } } }
I20260925 02:50:57.847432 10024 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } } }
I20260925 02:50:57.847929 10024 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:57.847805 9603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:57.848809 10029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } } }
I20260925 02:50:57.850430 10029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:50:57.850804 10028 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } } }
I20260925 02:50:57.851174 10028 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:57.851682 9841 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e197a1b4dfb94d7fb94f67e590716711 (127.2.172.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } } }
I20260925 02:50:57.854851 9869 sys_catalog.cc:455] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0fda7481e654efa9d6911ae0af27ec6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } } }
I20260925 02:50:57.855116 9869 sys_catalog.cc:458] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:50:57.875882 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 9233
I20260925 02:50:57.895493 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:33489
--local_ip_for_outbound_sockets=127.2.172.66
--tserver_master_addrs=127.2.172.126:36101
--webserver_port=36047
--webserver_interface=127.2.172.66
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab
--principal=oryx/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.172.126:36101,127.2.172.125:38125
--tserver_master_addrs=127.2.172.126:36101,127.2.172.125:38125,127.2.172.124:40083 with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:58.242046 10030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:58.242345 10030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:58.242514 10030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:58.250473 10030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:58.250600 10030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:58.250804 10030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:50:58.261198 10030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab
--principal=oryx/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:33489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=36047
--tserver_master_addrs=127.2.172.126:36101,127.2.172.125:38125,127.2.172.124:40083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:58.262658 10030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:58.264128 10030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:58.279263 10035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:58.279182 10036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:58.279230 10038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:58.280088 10030 server_base.cc:1034] running on GCE node
I20260925 02:50:58.280699 10030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:58.281633 10030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:58.282891 10030 hybrid_clock.cc:648] HybridClock initialized: now 1790304658282821 us; error 58 us; skew 500 ppm
I20260925 02:50:58.283241 10030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:58 dist-test-slave-g1xh krb5kdc[8991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304658, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:58.286572 10030 init.cc:377] Logged in from keytab as oryx/127.2.172.66@KRBTEST.COM (short username oryx)
I20260925 02:50:58.288295 10030 webserver.cc:460] Webserver started at http://127.2.172.66:36047/ using document root <none> and password file <none>
I20260925 02:50:58.289040 10030 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:58.289209 10030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:58.295493 10030 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:50:58.298619 10045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:58.300379 10030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:58.300545 10030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "79fab9fd748347559e7d7710682b465a"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-g1xh"
server_key: "0ade64b497265de5fddbdec14960f034"
server_key_iv: "1fea47ba5a76342eb15c62a6db4c0111"
server_key_version: "encryptionkey@0"
I20260925 02:50:58.301611 10030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:58.319845 10030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:58.324600 10030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:58.324963 10030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:58.326246 10030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:58.327728 10030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:58.327824 10030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:58.327913 10030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:58.327973 10030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:58.364431 10030 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:33489
I20260925 02:50:58.364527 10158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:33489 every 8 connection(s)
I20260925 02:50:58.365963 10030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:58.370460 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 10030
I20260925 02:50:58.370769 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 9369
Sep 25 02:50:58 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304658, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.124@KRBTEST.COM
Sep 25 02:50:58 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304658, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.125@KRBTEST.COM
Sep 25 02:50:58 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304658, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:50:58.407608 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:58.369882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:40083 (local address 127.2.172.66:40025)
0925 02:50:58.370913 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:58.370974 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:58.373940 (+ 2966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:58.382107 (+ 8167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:58.382124 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:58.383292 (+ 1168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:58.384396 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:58.384429 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.386752 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.386758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:58.388985 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:58.388999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.390254 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.390264 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:58.390630 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:58.392066 (+ 1436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:58.392116 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:58.397429 (+ 5313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:58.402234 (+ 4805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:58.402245 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:58.402281 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:58.402904 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:58.403420 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:58.403426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:58.403430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:58.403754 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:58.404563 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:58.404571 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:58.405122 (+ 551us) client_negotiation.cc:769] Sending connection context
0925 02:50:58.405529 (+ 407us) client_negotiation.cc:240] Negotiation successful
0925 02:50:58.406149 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":567,"spinlock_wait_cycles":28928,"thread_start_us":573,"threads_started":1}
Sep 25 02:50:58 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.172.67@KRBTEST.COM
I20260925 02:50:58.408661 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:58.369805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:40025 (local address 127.2.172.124:40083)
0925 02:50:58.381095 (+ 11290us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:58.381108 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:58.381142 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:50:58.381223 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:58.381252 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:58.381364 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:58.381507 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:58.385432 (+ 3925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.386567 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.389162 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.389784 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.389882 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:58.397650 (+ 7768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:58.397656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:58.397659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:58.397695 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:58.402082 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:58.403062 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:58.403068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:58.403072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:58.403269 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:58.403901 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:58.403906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:58.403909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:58.404317 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:58.404481 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:58.408156 (+ 3675us) server_negotiation.cc:299] Negotiation successful
0925 02:50:58.408369 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11129,"thread_start_us":11033,"threads_started":1}
I20260925 02:50:58.408941 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:58.369882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:38125 (local address 127.2.172.66:48057)
0925 02:50:58.377870 (+ 7988us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:58.377894 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:58.378022 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:58.385239 (+ 7217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:58.385243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:58.385886 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:58.386172 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:58.386183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.387620 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.387625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:58.389182 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:58.389193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.390499 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.390508 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:58.390721 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:58.392059 (+ 1338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:58.392088 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:58.396559 (+ 4471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:58.404055 (+ 7496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:58.404060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:58.404063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:58.404604 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:58.405147 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:58.405154 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:58.405161 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:58.405320 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:58.406005 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:58.406013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:58.406213 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 02:50:58.406403 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 02:50:58.406784 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7782,"mutex_wait_us":406,"thread_start_us":7175,"threads_started":1}
I20260925 02:50:58.410025 10159 heartbeater.cc:347] Connected to a master server at 127.2.172.124:40083
I20260925 02:50:58.410353 10159 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:58.410519 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:58.369421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:48057 (local address 127.2.172.125:38125)
0925 02:50:58.384949 (+ 15528us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:58.384953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:58.384969 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:50:58.385023 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:58.385026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:58.385083 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:58.385209 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:58.386350 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.387421 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.389342 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.389972 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.390089 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:58.398174 (+ 8085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:58.398185 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:58.398190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:58.398234 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:58.403890 (+ 5656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:58.404769 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:58.404774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:58.404777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:58.405011 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:58.405511 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:58.405515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:58.405519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:58.405870 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:58.406020 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:58.409854 (+ 3834us) server_negotiation.cc:299] Negotiation successful
0925 02:50:58.410150 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15401,"thread_start_us":15282,"threads_started":1}
I20260925 02:50:58.410727 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:58.378110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.66:43711)
0925 02:50:58.391239 (+ 13129us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:58.391262 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:58.392840 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:58.393423 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:58.393428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:58.394304 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:58.394571 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:58.394580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.395835 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.395848 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:58.397042 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:58.397060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.397998 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.398005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:58.398187 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:58.398843 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:58.398872 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:58.401333 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:58.407481 (+ 6148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:58.407487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:58.407492 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:58.408052 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:58.408679 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:58.408684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:58.408687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:58.408870 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:58.409656 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:58.409660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:58.409808 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 02:50:58.409973 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 02:50:58.410277 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12980,"thread_start_us":12805,"threads_started":1}
I20260925 02:50:58.411504 10161 heartbeater.cc:347] Connected to a master server at 127.2.172.125:38125
I20260925 02:50:58.411645 10161 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:58.412429 10161 heartbeater.cc:511] Master 127.2.172.125:38125 requested a full tablet report, sending...
I20260925 02:50:58.412459 10159 heartbeater.cc:511] Master 127.2.172.124:40083 requested a full tablet report, sending...
I20260925 02:50:58.413905 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:58.378251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:43711 (local address 127.2.172.126:36101)
0925 02:50:58.380710 (+ 2459us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:58.380719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:58.391370 (+ 10651us) server_negotiation.cc:407] Connection header received
0925 02:50:58.393027 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:58.393033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:58.393128 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:58.393209 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:58.394806 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.395667 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.397194 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.397806 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.397958 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:58.401540 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:58.401545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:58.401548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:58.401582 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:58.405497 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:58.408331 (+ 2834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:58.408337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:58.408341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:58.408536 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:58.409180 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:58.409186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:58.409188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:58.409504 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:58.409704 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:58.410092 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 02:50:58.410360 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2302,"thread_start_us":2194,"threads_started":1}
I20260925 02:50:58.414216 9642 ts_manager.cc:194] Registered new tserver with Master: 79fab9fd748347559e7d7710682b465a (127.2.172.66:33489)
I20260925 02:50:58.414201 9831 ts_manager.cc:194] Registered new tserver with Master: 79fab9fd748347559e7d7710682b465a (127.2.172.66:33489)
I20260925 02:50:58.415305 10160 heartbeater.cc:347] Connected to a master server at 127.2.172.126:36101
I20260925 02:50:58.415433 10160 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:58.415876 10160 heartbeater.cc:511] Master 127.2.172.126:36101 requested a full tablet report, sending...
I20260925 02:50:58.416839 9036 ts_manager.cc:194] Re-registered known tserver with Master: 79fab9fd748347559e7d7710682b465a (127.2.172.66:33489)
I20260925 02:50:58.417781 9036 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:43711
I20260925 02:50:58.419492 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:58.371753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.2.172.67:40383)
0925 02:50:58.391921 (+ 20168us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:58.391933 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:58.391967 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:50:58.392042 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:58.392051 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:58.392340 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:58.392524 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:58.396114 (+ 3590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.399673 (+ 3559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.401813 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:58.402838 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:58.402978 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:58.412140 (+ 9162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:58.412160 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:58.412175 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:58.412220 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:58.416351 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:58.417221 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:58.417230 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:58.417234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:58.417375 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:58.417798 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:58.417803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:58.417812 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:58.418202 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:58.418534 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:58.418853 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 02:50:58.419133 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19977,"thread_start_us":20138,"threads_started":1}
I20260925 02:50:58.420851 9493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:58.508381 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 9369
I20260925 02:50:58.524519 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:40383
--local_ip_for_outbound_sockets=127.2.172.67
--tserver_master_addrs=127.2.172.126:36101
--webserver_port=40883
--webserver_interface=127.2.172.67
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab
--principal=oryx/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.172.126:36101,127.2.172.125:38125
--tserver_master_addrs=127.2.172.126:36101,127.2.172.125:38125,127.2.172.124:40083 with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:50:58.778576 10012 heartbeater.cc:503] Master 127.2.172.126:36101 was elected leader, sending a full tablet report...
W20260925 02:50:58.868286 10172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:58.868673 10172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:58.868845 10172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:58.876596 10172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:58.876756 10172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:58.876936 10172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:50:58.887295 10172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab
--principal=oryx/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:40383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=40883
--tserver_master_addrs=127.2.172.126:36101,127.2.172.125:38125,127.2.172.124:40083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:50:58.888721 10172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:58.890086 10172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:58.904385 10177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:58.904541 10178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:58.904488 10180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:58.909634 10172 server_base.cc:1034] running on GCE node
I20260925 02:50:58.910406 10172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:58.911583 10172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:58.912868 10172 hybrid_clock.cc:648] HybridClock initialized: now 1790304658912822 us; error 42 us; skew 500 ppm
I20260925 02:50:58.913234 10172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:50:58 dist-test-slave-g1xh krb5kdc[8991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304658, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:58.916785 10172 init.cc:377] Logged in from keytab as oryx/127.2.172.67@KRBTEST.COM (short username oryx)
I20260925 02:50:58.918416 10172 webserver.cc:460] Webserver started at http://127.2.172.67:40883/ using document root <none> and password file <none>
I20260925 02:50:58.919166 10172 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:58.919332 10172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:58.925029 10172 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 02:50:58.927908 10187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:58.929328 10172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:50:58.929513 10172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "080f175629c54c96b845358d2413c7f7"
format_stamp: "Formatted at 2026-09-25 02:50:55 on dist-test-slave-g1xh"
server_key: "f6bf95e9072738f87b161536fd3d7c64"
server_key_iv: "daacfaf74f66d310edd3dd3394b2bed3"
server_key_version: "encryptionkey@0"
I20260925 02:50:58.930510 10172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:58.953559 10172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:58.958685 10172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:58.959043 10172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:58.960186 10172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:58.961722 10172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:58.961856 10172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:58.961998 10172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:58.962087 10172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:58.998131 10172 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:40383
I20260925 02:50:58.998174 10300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:40383 every 8 connection(s)
I20260925 02:50:58.999630 10172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:59.007987 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 10172
Sep 25 02:50:59 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304658, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.124@KRBTEST.COM
Sep 25 02:50:59 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304658, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.125@KRBTEST.COM
I20260925 02:50:59.053473 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.003663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:33929 (local address 127.2.172.125:38125)
0925 02:50:59.016629 (+ 12966us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.016636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.019616 (+ 2980us) server_negotiation.cc:407] Connection header received
0925 02:50:59.020249 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.020258 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.020382 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.020478 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:59.024546 (+ 4068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.026137 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.035670 (+ 9533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.036683 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.036812 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.044875 (+ 8063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:59.044882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:59.044890 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:59.044936 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:59.049082 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.050324 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.050330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.050335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.050529 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.051113 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.051119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.051122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.051489 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:59.051652 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.052787 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.053044 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12819,"spinlock_wait_cycles":3840,"thread_start_us":13351,"threads_started":1}
I20260925 02:50:59.053690 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.018767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.2.172.124:40083)
0925 02:50:59.022855 (+ 4088us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.022864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.022890 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:50:59.022969 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.022978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.023086 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.023221 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:59.030036 (+ 6815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.031343 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.032915 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.033649 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.033794 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.040182 (+ 6388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:59.040193 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:59.040199 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:59.040254 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:59.044058 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.049278 (+ 5220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.049283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.049286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.049500 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.050442 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.050448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.050451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.050822 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:59.051104 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.053061 (+ 1957us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.053313 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3922,"thread_start_us":3803,"threads_started":1}
I20260925 02:50:59.055409 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.002774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:50441 (local address 127.2.172.124:40083)
0925 02:50:59.015658 (+ 12884us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.015666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.024202 (+ 8536us) server_negotiation.cc:407] Connection header received
0925 02:50:59.029620 (+ 5418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.029630 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.029749 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.029846 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:59.031711 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.033181 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.035574 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.036312 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.036443 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.044887 (+ 8444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:59.044894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:59.044898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:59.044941 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:59.049094 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.052399 (+ 3305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.052405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.052408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.052561 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.053234 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.053239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.053243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.053621 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:59.053762 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.054746 (+ 984us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.054985 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12682,"thread_start_us":12545,"threads_started":1}
I20260925 02:50:59.057703 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.003385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:38125 (local address 127.2.172.67:33929)
0925 02:50:59.016845 (+ 13460us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:59.016900 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:59.019795 (+ 2895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:59.021205 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:59.021222 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:59.022211 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:59.023294 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.023343 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.026460 (+ 3117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.026468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.029332 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.029357 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.037227 (+ 7870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.037236 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.037528 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.038217 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:59.038259 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:59.044415 (+ 6156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:59.049278 (+ 4863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:59.049290 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:59.049315 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:59.050176 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:59.050688 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:59.050693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:59.050697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:59.050978 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:59.051765 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:59.051775 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:59.052327 (+ 552us) client_negotiation.cc:769] Sending connection context
0925 02:50:59.054265 (+ 1938us) client_negotiation.cc:240] Negotiation successful
0925 02:50:59.056260 (+ 1995us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12648,"mutex_wait_us":197,"thread_start_us":20153,"threads_started":1}
I20260925 02:50:59.057772 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.003385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:40083 (local address 127.2.172.67:50441)
0925 02:50:59.024092 (+ 20707us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:59.024117 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:59.024336 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:59.030177 (+ 5841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:59.030184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:59.031025 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:59.031471 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.031487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.033399 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.033407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.035355 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.035380 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.036779 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.036787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.037217 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.040359 (+ 3142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:59.040388 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:59.044620 (+ 4232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:59.049643 (+ 5023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:59.049647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:59.049651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:59.052227 (+ 2576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:59.052709 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:59.052714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:59.052717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:59.053088 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:59.053853 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:59.053860 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:59.054032 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 02:50:59.055684 (+ 1652us) client_negotiation.cc:240] Negotiation successful
0925 02:50:59.056148 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20497,"mutex_wait_us":452,"thread_start_us":11991,"threads_started":1}
I20260925 02:50:59.060127 10303 heartbeater.cc:347] Connected to a master server at 127.2.172.125:38125
I20260925 02:50:59.060565 10303 heartbeater.cc:464] Registering TS with master...
Sep 25 02:50:59 dist-test-slave-g1xh krb5kdc[8991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304658, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:50:59.061460 10303 heartbeater.cc:511] Master 127.2.172.125:38125 requested a full tablet report, sending...
I20260925 02:50:59.062582 10301 heartbeater.cc:347] Connected to a master server at 127.2.172.124:40083
I20260925 02:50:59.062723 10301 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:59.063191 10301 heartbeater.cc:511] Master 127.2.172.124:40083 requested a full tablet report, sending...
I20260925 02:50:59.064378 9642 ts_manager.cc:194] Registered new tserver with Master: 080f175629c54c96b845358d2413c7f7 (127.2.172.67:40383)
I20260925 02:50:59.064400 9831 ts_manager.cc:194] Registered new tserver with Master: 080f175629c54c96b845358d2413c7f7 (127.2.172.67:40383)
I20260925 02:50:59.065181 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.033677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.2.172.125:38125)
0925 02:50:59.043293 (+ 9616us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.043299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.043863 (+ 564us) server_negotiation.cc:407] Connection header received
0925 02:50:59.043932 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.043938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.044020 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.044185 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:59.046547 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.048354 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.050293 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.051015 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.051127 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.057148 (+ 6021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:59.057154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:59.057157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:59.057189 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:59.060895 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.061773 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.061778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.061782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.061958 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.062606 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.062612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.062616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.063007 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:59.063268 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.064491 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.064746 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9455,"thread_start_us":9322,"threads_started":1}
I20260925 02:50:59.070421 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.018123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.2.172.126:36101)
0925 02:50:59.027894 (+ 9771us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.027900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.034032 (+ 6132us) server_negotiation.cc:407] Connection header received
0925 02:50:59.034113 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.034118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.034212 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.034330 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:59.038134 (+ 3804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.039700 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.054538 (+ 14838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.055207 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.055340 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.063046 (+ 7706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:59.063053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:59.063059 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:59.063102 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:59.067133 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.068111 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.068120 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.068124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.068333 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.068788 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.068794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.068797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.069176 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:59.069451 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.069749 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.069949 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9632,"thread_start_us":30669,"threads_started":1}
I20260925 02:50:59.072414 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.024268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.67:58121)
0925 02:50:59.031284 (+ 7016us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:59.031324 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:59.031474 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:59.049000 (+ 17526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:59.049006 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:59.049745 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:59.050112 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.050125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.052809 (+ 2684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.052816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.055846 (+ 3030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.055863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.057815 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.057822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.058054 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.058604 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:59.058627 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:59.065933 (+ 7306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:59.069783 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:59.069789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:59.069792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:59.070322 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:59.070736 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:59.070741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:59.070744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:59.070966 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:59.071613 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:59.071616 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:59.071736 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 02:50:59.071923 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 02:50:59.072146 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6867,"thread_start_us":7303,"threads_started":1}
I20260925 02:50:59.072657 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.024372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:58121 (local address 127.2.172.126:36101)
0925 02:50:59.048209 (+ 23837us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.048215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.048234 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:50:59.048292 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.048297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.048453 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.048644 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:59.051381 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.052643 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.056059 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.056846 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.058363 (+ 1517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.066181 (+ 7818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:59.066188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:59.066193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:59.066238 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:59.069648 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.070459 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.070467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.070471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.070615 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.071106 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.071110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.071112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.071450 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:59.071596 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.072053 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.072263 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23685,"thread_start_us":3286,"threads_started":1}
I20260925 02:50:59.073549 10302 heartbeater.cc:347] Connected to a master server at 127.2.172.126:36101
I20260925 02:50:59.073709 10302 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:59.074185 10302 heartbeater.cc:511] Master 127.2.172.126:36101 requested a full tablet report, sending...
I20260925 02:50:59.075227 9035 ts_manager.cc:194] Re-registered known tserver with Master: 080f175629c54c96b845358d2413c7f7 (127.2.172.67:40383)
I20260925 02:50:59.076325 9035 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:58121
I20260925 02:50:59.075414 9036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:59.082985 9036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:59.125735 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.104125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:39097 (local address 127.0.0.1:37606)
0925 02:50:59.116501 (+ 12376us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:59.116526 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:59.116683 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:59.118703 (+ 2020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:59.118708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:59.118737 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:59.119812 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.119827 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.121372 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.121376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.123289 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.123302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.124758 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.124763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.124952 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.124994 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:50:59.125124 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 02:50:59.125274 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12219,"thread_start_us":12584,"threads_started":1}
I20260925 02:50:59.127911 10333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.104414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37606 (local address 127.2.172.65:39097)
0925 02:50:59.117981 (+ 13567us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.117997 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.118026 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:50:59.118199 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.118206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.118428 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.118722 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:59.120104 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.121154 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.123451 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.124548 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.124721 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.124962 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.125191 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.125356 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13407,"thread_start_us":13277,"threads_started":1}
I20260925 02:50:59.130280 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.104849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.2.172.67:40383)
0925 02:50:59.115853 (+ 11004us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.115868 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.120731 (+ 4863us) server_negotiation.cc:407] Connection header received
0925 02:50:59.120904 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.120918 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.121295 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.121568 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:59.123175 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.125425 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.127667 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.129186 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.129325 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.129556 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.129803 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.129968 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10742,"thread_start_us":10297,"threads_started":1}
I20260925 02:50:59.130300 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.104581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:40383 (local address 127.0.0.1:44116)
0925 02:50:59.117212 (+ 12631us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:59.117239 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:59.117419 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:59.121570 (+ 4151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:59.121575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:59.121614 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:59.122115 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.122151 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.125585 (+ 3434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.125590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.127448 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.127461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.129407 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.129414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.129604 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.129643 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:50:59.129755 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:50:59.129893 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12416,"thread_start_us":11566,"threads_started":1}
I20260925 02:50:59.133095 9945 tablet_service.cc:1467] Processing CreateTablet for tablet 2b80040a0e60497094d95b36b49b364b (DEFAULT_TABLE table=test-table [id=fd4f19f094864010824db4f0f5072668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:59.135146 9945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b80040a0e60497094d95b36b49b364b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:59.142023 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.117073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:33489 (local address 127.0.0.1:57410)
0925 02:50:59.126224 (+ 9151us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:59.126244 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:59.126361 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:59.135251 (+ 8890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:59.135257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:59.135299 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:59.135518 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.135532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.137022 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.137029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.138663 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.138685 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.140627 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.140633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.140849 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.140895 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:50:59.141431 (+ 536us) client_negotiation.cc:240] Negotiation successful
0925 02:50:59.141603 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9041,"thread_start_us":20103,"threads_started":1}
I20260925 02:50:59.141945 10236 tablet_service.cc:1467] Processing CreateTablet for tablet 2b80040a0e60497094d95b36b49b364b (DEFAULT_TABLE table=test-table [id=fd4f19f094864010824db4f0f5072668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:59.143105 10337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.117318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57410 (local address 127.2.172.66:33489)
0925 02:50:59.134683 (+ 17365us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.134690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.134720 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:50:59.134806 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.134812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.135012 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.135528 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:59.135802 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.136856 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.138866 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.140412 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.142290 (+ 1878us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.142520 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.142595 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.142779 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17172,"thread_start_us":17027,"threads_started":1}
I20260925 02:50:59.143752 10236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b80040a0e60497094d95b36b49b364b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:59.147531 10093 tablet_service.cc:1467] Processing CreateTablet for tablet 2b80040a0e60497094d95b36b49b364b (DEFAULT_TABLE table=test-table [id=fd4f19f094864010824db4f0f5072668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:59.149006 10093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b80040a0e60497094d95b36b49b364b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:59.151690 10338 tablet_bootstrap.cc:492] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2: Bootstrap starting.
I20260925 02:50:59.156067 10338 tablet_bootstrap.cc:654] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:59.157486 10338 log.cc:824] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:59.160900 10338 tablet_bootstrap.cc:492] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2: No bootstrap required, opened a new log
I20260925 02:50:59.161268 10340 tablet_bootstrap.cc:492] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7: Bootstrap starting.
I20260925 02:50:59.161293 10338 ts_tablet_manager.cc:1397] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260925 02:50:59.163075 10341 tablet_bootstrap.cc:492] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a: Bootstrap starting.
I20260925 02:50:59.165752 10340 tablet_bootstrap.cc:654] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:59.166836 10341 tablet_bootstrap.cc:654] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:59.167222 10340 log.cc:824] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:59.169121 10341 log.cc:824] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:59.170284 10338 raft_consensus.cc:348] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } } peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } }
I20260925 02:50:59.170941 10338 raft_consensus.cc:374] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:59.171145 10338 raft_consensus.cc:729] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ab128117884673903a3c39d748b7a2, State: Initialized, Role: FOLLOWER
I20260925 02:50:59.171845 10338 consensus_queue.cc:260] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } } peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } }
I20260925 02:50:59.173087 10341 tablet_bootstrap.cc:492] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a: No bootstrap required, opened a new log
I20260925 02:50:59.173422 10341 ts_tablet_manager.cc:1397] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 02:50:59.173484 10340 tablet_bootstrap.cc:492] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7: No bootstrap required, opened a new log
I20260925 02:50:59.173651 10338 ts_tablet_manager.cc:1428] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260925 02:50:59.173763 10340 ts_tablet_manager.cc:1397] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260925 02:50:59.185441 10340 raft_consensus.cc:348] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } } peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } }
I20260925 02:50:59.185946 10340 raft_consensus.cc:374] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:59.186058 10340 raft_consensus.cc:729] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080f175629c54c96b845358d2413c7f7, State: Initialized, Role: FOLLOWER
I20260925 02:50:59.186730 10340 consensus_queue.cc:260] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } } peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } }
I20260925 02:50:59.187031 10341 raft_consensus.cc:348] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } } peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } }
I20260925 02:50:59.187562 10341 raft_consensus.cc:374] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:59.187667 10341 raft_consensus.cc:729] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79fab9fd748347559e7d7710682b465a, State: Initialized, Role: FOLLOWER
I20260925 02:50:59.188359 10341 consensus_queue.cc:260] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } } peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } }
I20260925 02:50:59.188874 10340 ts_tablet_manager.cc:1428] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260925 02:50:59.189373 10302 heartbeater.cc:503] Master 127.2.172.126:36101 was elected leader, sending a full tablet report...
I20260925 02:50:59.189510 10160 heartbeater.cc:503] Master 127.2.172.126:36101 was elected leader, sending a full tablet report...
I20260925 02:50:59.190083 10341 ts_tablet_manager.cc:1428] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260925 02:50:59.224081 10014 tablet.cc:2369] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:50:59.258347 10304 tablet.cc:2369] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:59.358716 10344 raft_consensus.cc:482] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:59.359010 10344 raft_consensus.cc:504] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } } peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } }
I20260925 02:50:59.360853 10344 leader_election.cc:288] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 080f175629c54c96b845358d2413c7f7 (127.2.172.67:40383), 79fab9fd748347559e7d7710682b465a (127.2.172.66:33489)
I20260925 02:50:59.369410 10337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.362199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:42583 (local address 127.2.172.66:33489)
0925 02:50:59.362437 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.362442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.362781 (+ 339us) server_negotiation.cc:407] Connection header received
0925 02:50:59.362933 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.362937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.362999 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.363077 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:59.363769 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.364894 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.366771 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.368155 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.368330 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.368410 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.368959 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.369138 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 02:50:59.369722 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.361477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:50397 (local address 127.2.172.67:40383)
0925 02:50:59.361748 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.361755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.362138 (+ 383us) server_negotiation.cc:407] Connection header received
0925 02:50:59.362453 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.362460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.362545 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.362610 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:59.363487 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.364697 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.367325 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.368624 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.368789 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.368868 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.369332 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.369482 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:50:59.369846 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.361254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:40383 (local address 127.2.172.65:50397)
0925 02:50:59.362007 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:59.362034 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:59.362236 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:59.362838 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:59.362845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:59.362872 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:59.363330 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.363347 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.364862 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.364868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.367140 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.367157 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.368896 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.368903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.369128 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.369174 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:50:59.369286 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:50:59.369424 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":368,"threads_started":1}
I20260925 02:50:59.370249 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.361968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:33489 (local address 127.2.172.65:42583)
0925 02:50:59.362639 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:59.362655 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:59.362774 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:59.363119 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:59.363128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:59.363155 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:59.363534 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.363543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.365042 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.365057 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.366582 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:59.366599 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.368561 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.368568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:59.368749 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.368786 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:50:59.369614 (+ 828us) client_negotiation.cc:240] Negotiation successful
0925 02:50:59.369812 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":418,"threads_started":1}
I20260925 02:50:59.371306 10114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b80040a0e60497094d95b36b49b364b" candidate_uuid: "30ab128117884673903a3c39d748b7a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79fab9fd748347559e7d7710682b465a" is_pre_election: true
I20260925 02:50:59.371392 10256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b80040a0e60497094d95b36b49b364b" candidate_uuid: "30ab128117884673903a3c39d748b7a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "080f175629c54c96b845358d2413c7f7" is_pre_election: true
I20260925 02:50:59.371907 10114 raft_consensus.cc:2454] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ab128117884673903a3c39d748b7a2 in term 0.
I20260925 02:50:59.371940 10256 raft_consensus.cc:2454] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ab128117884673903a3c39d748b7a2 in term 0.
I20260925 02:50:59.372596 9899 leader_election.cc:302] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ab128117884673903a3c39d748b7a2, 79fab9fd748347559e7d7710682b465a; no voters:
I20260925 02:50:59.373129 10344 raft_consensus.cc:2781] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:59.373373 10344 raft_consensus.cc:482] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:59.373512 10344 raft_consensus.cc:3037] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:59.376895 10344 raft_consensus.cc:504] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } } peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } }
I20260925 02:50:59.377885 10344 leader_election.cc:288] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [CANDIDATE]: Term 1 election: Requested vote from peers 080f175629c54c96b845358d2413c7f7 (127.2.172.67:40383), 79fab9fd748347559e7d7710682b465a (127.2.172.66:33489)
I20260925 02:50:59.378337 10114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b80040a0e60497094d95b36b49b364b" candidate_uuid: "30ab128117884673903a3c39d748b7a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79fab9fd748347559e7d7710682b465a"
I20260925 02:50:59.378448 10256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b80040a0e60497094d95b36b49b364b" candidate_uuid: "30ab128117884673903a3c39d748b7a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "080f175629c54c96b845358d2413c7f7"
I20260925 02:50:59.378569 10114 raft_consensus.cc:3037] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:59.378777 10256 raft_consensus.cc:3037] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7 [term 0 FOLLOWER]: Advancing to term 1
W20260925 02:50:59.379918 10162 tablet.cc:2369] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:59.381508 10114 raft_consensus.cc:2454] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ab128117884673903a3c39d748b7a2 in term 1.
I20260925 02:50:59.381805 10256 raft_consensus.cc:2454] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ab128117884673903a3c39d748b7a2 in term 1.
I20260925 02:50:59.381996 9899 leader_election.cc:302] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ab128117884673903a3c39d748b7a2, 79fab9fd748347559e7d7710682b465a; no voters:
I20260925 02:50:59.382315 10344 raft_consensus.cc:2781] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:59.382826 10344 raft_consensus.cc:686] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 1 LEADER]: Becoming Leader. State: Replica: 30ab128117884673903a3c39d748b7a2, State: Running, Role: LEADER
I20260925 02:50:59.383380 10344 consensus_queue.cc:237] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } } peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } }
I20260925 02:50:59.388788 9035 catalog_manager.cc:5495] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ab128117884673903a3c39d748b7a2 (127.2.172.65). New cstate: current_term: 1 leader_uuid: "30ab128117884673903a3c39d748b7a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } health_report { overall_health: UNKNOWN } } }
I20260925 02:50:59.446041 10333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.440689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.2.172.65:39097)
0925 02:50:59.440948 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.440953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.440967 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:50:59.441057 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.441060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.441134 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.441296 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:59.441675 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.442389 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.443820 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.444241 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.444377 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.444845 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:59.445347 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:59.445521 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.445659 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.445740 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 02:50:59.454623 10256 raft_consensus.cc:1261] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7 [term 1 FOLLOWER]: Refusing update from remote peer 30ab128117884673903a3c39d748b7a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:59.454623 10114 raft_consensus.cc:1261] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a [term 1 FOLLOWER]: Refusing update from remote peer 30ab128117884673903a3c39d748b7a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:59.455786 10349 consensus_queue.cc:1035] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:59.456313 10344 consensus_queue.cc:1035] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:59.474547 10352 mvcc.cc:205] Tried to move back new op lower bound from 7333087885115326464 to 7333087884838420480. Current Snapshot: MvccSnapshot[applied={T|T < 7333087885115326464}]
I20260925 02:50:59.475006 10353 mvcc.cc:205] Tried to move back new op lower bound from 7333087885115326464 to 7333087884838420480. Current Snapshot: MvccSnapshot[applied={T|T < 7333087885115326464}]
I20260925 02:50:59.479996 10354 mvcc.cc:205] Tried to move back new op lower bound from 7333087885115326464 to 7333087884838420480. Current Snapshot: MvccSnapshot[applied={T|T < 7333087885115326464}]
I20260925 02:50:59.512029 9035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:59.512398 9035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:59.519337 9035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0fda7481e654efa9d6911ae0af27ec6: Sending DeleteTablet for 3 replicas of tablet 2b80040a0e60497094d95b36b49b364b
I20260925 02:50:59.520696 9945 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b80040a0e60497094d95b36b49b364b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:59 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:37606
I20260925 02:50:59.521400 10236 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b80040a0e60497094d95b36b49b364b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:59 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:44116
I20260925 02:50:59.521489 10093 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b80040a0e60497094d95b36b49b364b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:59 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:57410
I20260925 02:50:59.532352 10364 tablet_replica.cc:330] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a: stopping tablet replica
I20260925 02:50:59.533038 10364 raft_consensus.cc:2229] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:59.533726 10364 raft_consensus.cc:2258] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:59.536193 10363 tablet_replica.cc:330] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7: stopping tablet replica
I20260925 02:50:59.536660 10363 raft_consensus.cc:2229] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:59.537508 10362 tablet_replica.cc:330] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2: stopping tablet replica
I20260925 02:50:59.537946 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 9004
I20260925 02:50:59.537947 10362 raft_consensus.cc:2229] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:50:59.538690 10362 raft_consensus.cc:2258] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:59.539420 10363 raft_consensus.cc:2258] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:59.540709 9066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:50:59.542325 10364 ts_tablet_manager.cc:1905] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:59.554499 10363 ts_tablet_manager.cc:1905] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:59.556540 10362 ts_tablet_manager.cc:1905] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:59.559440 10364 ts_tablet_manager.cc:1918] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:59.559681 10364 log.cc:1196] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/2b80040a0e60497094d95b36b49b364b
I20260925 02:50:59.560314 10364 ts_tablet_manager.cc:1939] T 2b80040a0e60497094d95b36b49b364b P 79fab9fd748347559e7d7710682b465a: Deleting consensus metadata
I20260925 02:50:59.564055 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.524047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.2.172.125:38125)
0925 02:50:59.524248 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.524255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.535987 (+ 11732us) server_negotiation.cc:407] Connection header received
0925 02:50:59.536205 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.536211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.536305 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.536423 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:59.541858 (+ 5435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.543359 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.548960 (+ 5601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.549661 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.549779 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.555761 (+ 5982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:59.555770 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:59.555776 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:59.555833 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:59.559441 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.561467 (+ 2026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.561474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.561478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.561647 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.562151 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.562156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.562159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.562526 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:59.562777 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.563372 (+ 595us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.563641 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:50:59.572474 10362 ts_tablet_manager.cc:1918] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:59.572702 10362 log.cc:1196] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/2b80040a0e60497094d95b36b49b364b
I20260925 02:50:59.573407 10362 ts_tablet_manager.cc:1939] T 2b80040a0e60497094d95b36b49b364b P 30ab128117884673903a3c39d748b7a2: Deleting consensus metadata
I20260925 02:50:59.579411 10363 ts_tablet_manager.cc:1918] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:59.579672 10363 log.cc:1196] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/2b80040a0e60497094d95b36b49b364b
I20260925 02:50:59.580351 10363 ts_tablet_manager.cc:1939] T 2b80040a0e60497094d95b36b49b364b P 080f175629c54c96b845358d2413c7f7: Deleting consensus metadata
I20260925 02:50:59.583658 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.536822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.2.172.124:40083)
0925 02:50:59.536992 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.536997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.565465 (+ 28468us) server_negotiation.cc:407] Connection header received
0925 02:50:59.565621 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.565626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.565711 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.565814 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:59.567714 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.568975 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.571040 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.571697 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.571801 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.575468 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:59.575478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:59.575486 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:59.575539 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:59.579595 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.580746 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.580754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.580759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.580947 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.581472 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.581476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.581479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.581893 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:59.582130 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.582904 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.583180 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:50:59.689075 9022 catalog_manager.cc:4841] TS 080f175629c54c96b845358d2413c7f7 (127.2.172.67:40383): tablet 2b80040a0e60497094d95b36b49b364b (table test-table [id=fd4f19f094864010824db4f0f5072668]) successfully deleted
I20260925 02:50:59.691653 9020 catalog_manager.cc:4841] TS 30ab128117884673903a3c39d748b7a2 (127.2.172.65:39097): tablet 2b80040a0e60497094d95b36b49b364b (table test-table [id=fd4f19f094864010824db4f0f5072668]) successfully deleted
I20260925 02:50:59.692780 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 9004
I20260925 02:50:59.754372 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.723027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.2.172.124:40083)
0925 02:50:59.723270 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.723279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.736721 (+ 13442us) server_negotiation.cc:407] Connection header received
0925 02:50:59.736981 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.736987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.737067 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.737134 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:59.739329 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.740478 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.742504 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.743185 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.743282 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.747282 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:59.747289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:59.747295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:59.747346 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:59.751093 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.751991 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.751997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.752002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.752149 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.752613 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.752619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.752622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.752985 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:59.753241 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.753796 (+ 555us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.754021 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 02:50:59.758639 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:59.740439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.2.172.125:38125)
0925 02:50:59.740629 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:59.740636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:59.740858 (+ 222us) server_negotiation.cc:407] Connection header received
0925 02:50:59.741065 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:59.741071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:59.741168 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:59.741291 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:59.743727 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.744881 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.747434 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:59.748145 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:59.748302 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:59.751639 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:59.751647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:59.751653 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:59.751694 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:59.755525 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.756473 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.756479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.756484 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.756642 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:59.757119 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:59.757124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:59.757128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:59.757445 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:59.757685 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:59.758077 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 02:50:59.758292 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:51:00.484933 10373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:00.483917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.66:56343)
0925 02:51:00.484517 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:00.484683 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":309,"threads_started":1}
W20260925 02:51:00.485499 10160 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:36101 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
I20260925 02:51:00.499478 10374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:00.498603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.65:56321)
0925 02:51:00.499136 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:00.499282 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":286,"threads_started":1}
W20260925 02:51:00.500101 10012 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:36101 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
I20260925 02:51:00.506742 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:00.505653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.67:46381)
0925 02:51:00.506372 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:00.506530 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":406,"threads_started":1}
W20260925 02:51:00.507287 10302 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:36101 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
I20260925 02:51:01.020232 10376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e0fda7481e654efa9d6911ae0af27ec6)
I20260925 02:51:01.020627 10376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } }
I20260925 02:51:01.020985 10377 raft_consensus.cc:482] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e0fda7481e654efa9d6911ae0af27ec6)
I20260925 02:51:01.021323 10377 raft_consensus.cc:504] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } }
I20260925 02:51:01.021945 10376 leader_election.cc:288] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0fda7481e654efa9d6911ae0af27ec6 (127.2.172.126:36101), e197a1b4dfb94d7fb94f67e590716711 (127.2.172.124:40083)
I20260925 02:51:01.023967 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.022177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39846)
0925 02:51:01.023175 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.023512 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":777,"threads_started":1}
W20260925 02:51:01.025581 9629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
I20260925 02:51:01.028091 10377 leader_election.cc:288] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0fda7481e654efa9d6911ae0af27ec6 (127.2.172.126:36101), 3db79e82ab784ac3845a9e300581eea2 (127.2.172.125:38125)
I20260925 02:51:01.028293 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.027533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39854)
0925 02:51:01.027781 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.027993 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:51:01.028574 9629 leader_election.cc:334] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e0fda7481e654efa9d6911ae0af27ec6 (127.2.172.126:36101): Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
I20260925 02:51:01.029479 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.023338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39850)
0925 02:51:01.028720 (+ 5382us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.029040 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5253,"thread_start_us":5091,"threads_started":1}
W20260925 02:51:01.030053 9817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
I20260925 02:51:01.033061 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.031775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39860)
0925 02:51:01.032420 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.032701 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":324,"threads_started":1}
W20260925 02:51:01.033478 9817 leader_election.cc:334] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e0fda7481e654efa9d6911ae0af27ec6 (127.2.172.126:36101): Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
I20260925 02:51:01.034801 10380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.022648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.2.172.124:40083)
0925 02:51:01.025777 (+ 3129us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:01.025786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:01.025812 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:51:01.025928 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:01.025934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:01.026054 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:01.027007 (+ 953us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:01.027255 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.028816 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.031642 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.033109 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.033307 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.033487 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:01.034158 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 02:51:01.034324 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":2975,"thread_start_us":2783,"threads_started":1}
I20260925 02:51:01.034832 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.022190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:40083 (local address 127.0.0.1:52864)
0925 02:51:01.024462 (+ 2272us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.024503 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:01.024741 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:01.026335 (+ 1594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:01.026345 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:01.026412 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:01.026798 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:01.026817 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.029089 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.029096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:01.031440 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:01.031454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.033674 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.033682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:01.033914 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.033970 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 02:51:01.034128 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 02:51:01.034312 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2103,"mutex_wait_us":47,"thread_start_us":535,"threads_started":1}
I20260925 02:51:01.035784 9841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3db79e82ab784ac3845a9e300581eea2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e197a1b4dfb94d7fb94f67e590716711" is_pre_election: true
I20260925 02:51:01.036243 9841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db79e82ab784ac3845a9e300581eea2 in term 1.
I20260925 02:51:01.037052 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.028949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.2.172.125:38125)
0925 02:51:01.029764 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:01.029797 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:01.030037 (+ 240us) server_negotiation.cc:407] Connection header received
0925 02:51:01.030270 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:01.030277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:01.030361 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:01.030497 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:01.031169 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.032189 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.034558 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.035929 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.036085 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.036196 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:01.036659 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 02:51:01.036762 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":346,"threads_started":1}
I20260925 02:51:01.037079 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.029587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:38125 (local address 127.0.0.1:44032)
0925 02:51:01.029925 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.029941 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:01.030113 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:01.030520 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:01.030526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:01.030593 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:01.030927 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:01.030947 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.032389 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.032397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:01.034398 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:01.034411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.036185 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.036199 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:01.036386 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.036492 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 02:51:01.036615 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 02:51:01.036770 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":139}
I20260925 02:51:01.036981 9628 leader_election.cc:302] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db79e82ab784ac3845a9e300581eea2, e197a1b4dfb94d7fb94f67e590716711; no voters: e0fda7481e654efa9d6911ae0af27ec6
I20260925 02:51:01.037623 10376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:51:01.037801 10376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e0fda7481e654efa9d6911ae0af27ec6)
I20260925 02:51:01.037968 10376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:51:01.037966 9651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e197a1b4dfb94d7fb94f67e590716711" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3db79e82ab784ac3845a9e300581eea2" is_pre_election: true
I20260925 02:51:01.040791 10376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } }
I20260925 02:51:01.041317 9651 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e197a1b4dfb94d7fb94f67e590716711 in current term 2: Already voted for candidate 3db79e82ab784ac3845a9e300581eea2 in this term.
I20260925 02:51:01.041692 10376 leader_election.cc:288] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [CANDIDATE]: Term 2 election: Requested vote from peers e0fda7481e654efa9d6911ae0af27ec6 (127.2.172.126:36101), e197a1b4dfb94d7fb94f67e590716711 (127.2.172.124:40083)
I20260925 02:51:01.042338 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.041753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39876)
0925 02:51:01.042001 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.042126 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260925 02:51:01.042232 9841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3db79e82ab784ac3845a9e300581eea2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e197a1b4dfb94d7fb94f67e590716711"
I20260925 02:51:01.042500 9841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:51:01.042644 9817 leader_election.cc:302] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e197a1b4dfb94d7fb94f67e590716711; no voters: 3db79e82ab784ac3845a9e300581eea2, e0fda7481e654efa9d6911ae0af27ec6
I20260925 02:51:01.043433 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.043052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39884)
0925 02:51:01.043208 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.043315 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 02:51:01.043651 9629 leader_election.cc:334] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e0fda7481e654efa9d6911ae0af27ec6 (127.2.172.126:36101): Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
I20260925 02:51:01.045169 9841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db79e82ab784ac3845a9e300581eea2 in term 2.
I20260925 02:51:01.045372 10377 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:51:01.045723 9628 leader_election.cc:302] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db79e82ab784ac3845a9e300581eea2, e197a1b4dfb94d7fb94f67e590716711; no voters: e0fda7481e654efa9d6911ae0af27ec6
I20260925 02:51:01.046084 10376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:51:01.046828 10376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [term 2 LEADER]: Becoming Leader. State: Replica: 3db79e82ab784ac3845a9e300581eea2, State: Running, Role: LEADER
I20260925 02:51:01.047343 10376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } }
I20260925 02:51:01.049976 10387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3db79e82ab784ac3845a9e300581eea2. Latest consensus state: current_term: 2 leader_uuid: "3db79e82ab784ac3845a9e300581eea2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } } }
I20260925 02:51:01.050513 10387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:01.051280 10389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:01.057343 10389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:01.058522 10389 catalog_manager.cc:1259] Loaded cluster ID: d9373e52acd046568f4e4682f8104893
I20260925 02:51:01.058655 10389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:01.060226 10389 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:01.061250 10389 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: Loaded TSK: 0
I20260925 02:51:01.062382 10389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:51:01.124850 9642 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:51:01.133332 9642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:51:01.138989 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.138439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39910)
0925 02:51:01.138646 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.138768 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 02:51:01.139245 9841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [term 2 FOLLOWER]: Refusing update from remote peer 3db79e82ab784ac3845a9e300581eea2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 02:51:01.140285 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.139844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39918)
0925 02:51:01.140024 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.140146 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 02:51:01.140209 10387 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 02:51:01.140703 9629 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 -> Peer e0fda7481e654efa9d6911ae0af27ec6 (127.2.172.126:36101): Couldn't send request to peer e0fda7481e654efa9d6911ae0af27ec6. Status: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:51:01.143955 10377 sys_catalog.cc:455] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3db79e82ab784ac3845a9e300581eea2. Latest consensus state: current_term: 2 leader_uuid: "3db79e82ab784ac3845a9e300581eea2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } } }
I20260925 02:51:01.144345 10377 sys_catalog.cc:458] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:51:01.145946 10387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3db79e82ab784ac3845a9e300581eea2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } } }
I20260925 02:51:01.146366 10387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:01.147101 10377 sys_catalog.cc:455] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3db79e82ab784ac3845a9e300581eea2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e0fda7481e654efa9d6911ae0af27ec6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36101 } } peers { permanent_uuid: "3db79e82ab784ac3845a9e300581eea2" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 38125 } attrs { promote: false } } peers { permanent_uuid: "e197a1b4dfb94d7fb94f67e590716711" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 40083 } attrs { promote: false } } }
I20260925 02:51:01.147446 10377 sys_catalog.cc:458] T 00000000000000000000000000000000 P e197a1b4dfb94d7fb94f67e590716711 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:51:01.165724 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.156540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.2.172.67:40383)
0925 02:51:01.157092 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:01.157099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:01.157118 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:51:01.159158 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:01.159165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:01.159299 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:01.159433 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:01.159980 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.161172 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.163601 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.164629 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.164792 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.164923 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:01.165213 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 02:51:01.165373 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":295,"threads_started":1}
I20260925 02:51:01.165787 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.156375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:40383 (local address 127.0.0.1:44130)
0925 02:51:01.156643 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.156667 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:01.156792 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:01.159469 (+ 2677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:01.159474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:01.159500 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:01.159763 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:01.159775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.161320 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.161326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:01.163388 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:01.163404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.164834 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.164840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:01.165029 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.165064 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 02:51:01.165180 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 02:51:01.165348 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":49}
I20260925 02:51:01.166261 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.157155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:39097 (local address 127.0.0.1:37624)
0925 02:51:01.157645 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.157677 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:01.157793 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:01.159236 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:01.159241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:01.159267 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:01.159710 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:01.159722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.160799 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.160806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:01.162959 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:01.162974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.164937 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.164944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:01.165135 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.165175 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 02:51:01.165251 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:51:01.165418 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330}
I20260925 02:51:01.166625 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.157163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:33489 (local address 127.0.0.1:57418)
0925 02:51:01.157933 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.157952 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:01.158105 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:01.159666 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:01.159677 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:01.159707 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:01.160151 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:01.160162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.161579 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.161587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:01.163787 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:01.163801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.165707 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.165714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:01.165899 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.165947 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 02:51:01.166053 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 02:51:01.166180 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":80,"thread_start_us":379,"threads_started":1}
I20260925 02:51:01.166810 10236 tablet_service.cc:1467] Processing CreateTablet for tablet 81cccf5f6f29459087adc17c9076ef51 (DEFAULT_TABLE table=test-table [id=e9025fc3b17a4a0da345928d2c9c33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:01.167433 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.157294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.2.172.66:33489)
0925 02:51:01.159139 (+ 1845us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:01.159147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:01.159169 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:51:01.159253 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:01.159258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:01.159348 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:01.159459 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:01.160344 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.161444 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.163955 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.165282 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.165405 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.165534 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:01.166917 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0925 02:51:01.167123 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1691,"thread_start_us":1585,"threads_started":1}
I20260925 02:51:01.167433 10236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cccf5f6f29459087adc17c9076ef51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:01.168093 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.158063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.2.172.65:39097)
0925 02:51:01.158796 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:01.158805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:01.158829 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:51:01.158927 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:01.158932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:01.159018 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:01.159201 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:01.159875 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.160655 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.163358 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.164653 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.167117 (+ 2464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.167227 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:01.167285 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 02:51:01.167467 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":470,"threads_started":1}
I20260925 02:51:01.169205 9945 tablet_service.cc:1467] Processing CreateTablet for tablet 81cccf5f6f29459087adc17c9076ef51 (DEFAULT_TABLE table=test-table [id=e9025fc3b17a4a0da345928d2c9c33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:01.169783 9945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cccf5f6f29459087adc17c9076ef51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:01.172720 10093 tablet_service.cc:1467] Processing CreateTablet for tablet 81cccf5f6f29459087adc17c9076ef51 (DEFAULT_TABLE table=test-table [id=e9025fc3b17a4a0da345928d2c9c33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:01.173233 10093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cccf5f6f29459087adc17c9076ef51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:01.174140 10400 tablet_bootstrap.cc:492] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7: Bootstrap starting.
I20260925 02:51:01.175940 10400 tablet_bootstrap.cc:654] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:01.176069 10401 tablet_bootstrap.cc:492] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2: Bootstrap starting.
I20260925 02:51:01.177874 10401 tablet_bootstrap.cc:654] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:01.178238 10400 tablet_bootstrap.cc:492] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7: No bootstrap required, opened a new log
I20260925 02:51:01.178424 10400 ts_tablet_manager.cc:1397] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 02:51:01.180245 10400 raft_consensus.cc:348] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } } peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } }
I20260925 02:51:01.180521 10400 raft_consensus.cc:374] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:01.180631 10400 raft_consensus.cc:729] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080f175629c54c96b845358d2413c7f7, State: Initialized, Role: FOLLOWER
I20260925 02:51:01.181006 10401 tablet_bootstrap.cc:492] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2: No bootstrap required, opened a new log
I20260925 02:51:01.180884 10400 consensus_queue.cc:260] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } } peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } }
I20260925 02:51:01.181149 10401 ts_tablet_manager.cc:1397] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260925 02:51:01.181717 10400 ts_tablet_manager.cc:1428] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:51:01.182000 10401 raft_consensus.cc:348] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } } peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } }
I20260925 02:51:01.182233 10401 raft_consensus.cc:374] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:01.182353 10401 raft_consensus.cc:729] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ab128117884673903a3c39d748b7a2, State: Initialized, Role: FOLLOWER
I20260925 02:51:01.182578 10401 consensus_queue.cc:260] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } } peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } }
I20260925 02:51:01.183801 10401 ts_tablet_manager.cc:1428] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 02:51:01.184013 10404 tablet_bootstrap.cc:492] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a: Bootstrap starting.
I20260925 02:51:01.185814 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.184622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.65:48685)
0925 02:51:01.185295 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.185494 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"threads_started":1}
I20260925 02:51:01.186031 10404 tablet_bootstrap.cc:654] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:01.187901 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.182056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.67:55977)
0925 02:51:01.187449 (+ 5393us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.187614 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5227,"thread_start_us":5073,"threads_started":1}
I20260925 02:51:01.189556 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.188502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.67:33693)
0925 02:51:01.189264 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.189373 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":660,"threads_started":1}
I20260925 02:51:01.192354 10404 tablet_bootstrap.cc:492] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a: No bootstrap required, opened a new log
I20260925 02:51:01.192530 10404 ts_tablet_manager.cc:1397] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260925 02:51:01.193264 10013 heartbeater.cc:503] Master 127.2.172.125:38125 was elected leader, sending a full tablet report...
I20260925 02:51:01.193516 10404 raft_consensus.cc:348] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } } peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } }
I20260925 02:51:01.193701 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.192169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.65:36039)
0925 02:51:01.193378 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.193529 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1094}
I20260925 02:51:01.193751 10404 raft_consensus.cc:374] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:01.193763 10303 heartbeater.cc:503] Master 127.2.172.125:38125 was elected leader, sending a full tablet report...
I20260925 02:51:01.193849 10404 raft_consensus.cc:729] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79fab9fd748347559e7d7710682b465a, State: Initialized, Role: FOLLOWER
I20260925 02:51:01.194128 10404 consensus_queue.cc:260] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } } peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } }
I20260925 02:51:01.195075 10404 ts_tablet_manager.cc:1428] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:51:01.197547 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.196418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.66:58791)
0925 02:51:01.197056 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.197244 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":288,"threads_started":1}
I20260925 02:51:01.198416 10161 heartbeater.cc:503] Master 127.2.172.125:38125 was elected leader, sending a full tablet report...
I20260925 02:51:01.400655 10407 raft_consensus.cc:482] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:01.400920 10407 raft_consensus.cc:504] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } } peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } }
I20260925 02:51:01.401594 10407 leader_election.cc:288] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 080f175629c54c96b845358d2413c7f7 (127.2.172.67:40383), 79fab9fd748347559e7d7710682b465a (127.2.172.66:33489)
I20260925 02:51:01.402264 10256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cccf5f6f29459087adc17c9076ef51" candidate_uuid: "30ab128117884673903a3c39d748b7a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "080f175629c54c96b845358d2413c7f7" is_pre_election: true
I20260925 02:51:01.402280 10114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cccf5f6f29459087adc17c9076ef51" candidate_uuid: "30ab128117884673903a3c39d748b7a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79fab9fd748347559e7d7710682b465a" is_pre_election: true
I20260925 02:51:01.402526 10114 raft_consensus.cc:2454] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ab128117884673903a3c39d748b7a2 in term 0.
I20260925 02:51:01.402522 10256 raft_consensus.cc:2454] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ab128117884673903a3c39d748b7a2 in term 0.
I20260925 02:51:01.403015 9899 leader_election.cc:302] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ab128117884673903a3c39d748b7a2, 79fab9fd748347559e7d7710682b465a; no voters:
I20260925 02:51:01.403347 10407 raft_consensus.cc:2781] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:01.403491 10407 raft_consensus.cc:482] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:01.403625 10407 raft_consensus.cc:3037] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:01.405795 10407 raft_consensus.cc:504] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } } peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } }
I20260925 02:51:01.406342 10407 leader_election.cc:288] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [CANDIDATE]: Term 1 election: Requested vote from peers 080f175629c54c96b845358d2413c7f7 (127.2.172.67:40383), 79fab9fd748347559e7d7710682b465a (127.2.172.66:33489)
I20260925 02:51:01.407039 10114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cccf5f6f29459087adc17c9076ef51" candidate_uuid: "30ab128117884673903a3c39d748b7a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79fab9fd748347559e7d7710682b465a"
I20260925 02:51:01.407034 10256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cccf5f6f29459087adc17c9076ef51" candidate_uuid: "30ab128117884673903a3c39d748b7a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "080f175629c54c96b845358d2413c7f7"
I20260925 02:51:01.407238 10256 raft_consensus.cc:3037] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:01.407254 10114 raft_consensus.cc:3037] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:01.409310 10114 raft_consensus.cc:2454] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ab128117884673903a3c39d748b7a2 in term 1.
I20260925 02:51:01.409324 10256 raft_consensus.cc:2454] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ab128117884673903a3c39d748b7a2 in term 1.
I20260925 02:51:01.409778 9899 leader_election.cc:302] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 080f175629c54c96b845358d2413c7f7, 30ab128117884673903a3c39d748b7a2; no voters:
I20260925 02:51:01.410084 10407 raft_consensus.cc:2781] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:01.410528 10407 raft_consensus.cc:686] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 1 LEADER]: Becoming Leader. State: Replica: 30ab128117884673903a3c39d748b7a2, State: Running, Role: LEADER
I20260925 02:51:01.410876 10407 consensus_queue.cc:237] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } } peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } }
I20260925 02:51:01.411681 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.411122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.65:55339)
0925 02:51:01.411369 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.411465 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260925 02:51:01.413838 9641 catalog_manager.cc:5495] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ab128117884673903a3c39d748b7a2 (127.2.172.65). New cstate: current_term: 1 leader_uuid: "30ab128117884673903a3c39d748b7a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ab128117884673903a3c39d748b7a2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39097 } health_report { overall_health: HEALTHY } } }
I20260925 02:51:01.466760 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.460689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.2.172.65:39097)
0925 02:51:01.460872 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:01.460877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:01.460896 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:51:01.461118 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:01.461122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:01.461202 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:01.461311 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:01.461719 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.462467 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.464674 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.465209 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.465328 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.465854 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:01.465972 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:01.466049 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:01.466291 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 02:51:01.466368 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:51:01.470439 10256 raft_consensus.cc:1261] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7 [term 1 FOLLOWER]: Refusing update from remote peer 30ab128117884673903a3c39d748b7a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:01.470662 10114 raft_consensus.cc:1261] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a [term 1 FOLLOWER]: Refusing update from remote peer 30ab128117884673903a3c39d748b7a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:01.471288 10413 consensus_queue.cc:1035] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "080f175629c54c96b845358d2413c7f7" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:01.471692 10407 consensus_queue.cc:1035] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79fab9fd748347559e7d7710682b465a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:01.474731 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.473864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.66:59159)
0925 02:51:01.474430 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.474540 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420}
I20260925 02:51:01.475407 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.474563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.67:49889)
0925 02:51:01.474882 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.475111 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260925 02:51:01.484848 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.484252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.65:56097)
0925 02:51:01.484540 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.484644 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260925 02:51:01.485019 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.484447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.66:42829)
0925 02:51:01.484666 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.484788 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260925 02:51:01.491153 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.490568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.2.172.67:45895)
0925 02:51:01.490841 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.490947 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260925 02:51:01.493454 9642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:01.493710 9642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:01.500190 9642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3db79e82ab784ac3845a9e300581eea2: Sending DeleteTablet for 3 replicas of tablet 81cccf5f6f29459087adc17c9076ef51
I20260925 02:51:01.501330 10236 tablet_service.cc:1514] Processing DeleteTablet for tablet 81cccf5f6f29459087adc17c9076ef51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:01 UTC) from {username='oryx', principal='oryx/127.2.172.125@KRBTEST.COM'} at 127.0.0.1:44130
I20260925 02:51:01.501405 10093 tablet_service.cc:1514] Processing DeleteTablet for tablet 81cccf5f6f29459087adc17c9076ef51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:01 UTC) from {username='oryx', principal='oryx/127.2.172.125@KRBTEST.COM'} at 127.0.0.1:57418
I20260925 02:51:01.501886 9945 tablet_service.cc:1514] Processing DeleteTablet for tablet 81cccf5f6f29459087adc17c9076ef51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:01 UTC) from {username='oryx', principal='oryx/127.2.172.125@KRBTEST.COM'} at 127.0.0.1:37624
I20260925 02:51:01.502193 10424 tablet_replica.cc:330] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7: stopping tablet replica
I20260925 02:51:01.502475 10424 raft_consensus.cc:2229] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:01.502779 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 9818
I20260925 02:51:01.502959 10424 raft_consensus.cc:2258] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:01.503679 10425 tablet_replica.cc:330] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a: stopping tablet replica
I20260925 02:51:01.504005 10425 raft_consensus.cc:2229] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:01.504415 10425 raft_consensus.cc:2258] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:01.505733 10425 ts_tablet_manager.cc:1905] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:01.507823 10424 ts_tablet_manager.cc:1905] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:01.511380 10425 ts_tablet_manager.cc:1918] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:01.511626 10425 log.cc:1196] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/81cccf5f6f29459087adc17c9076ef51
I20260925 02:51:01.512167 10425 ts_tablet_manager.cc:1939] T 81cccf5f6f29459087adc17c9076ef51 P 79fab9fd748347559e7d7710682b465a: Deleting consensus metadata
I20260925 02:51:01.512352 10426 tablet_replica.cc:330] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2: stopping tablet replica
I20260925 02:51:01.512692 10426 raft_consensus.cc:2229] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:51:01.513221 10426 raft_consensus.cc:2258] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:01.513568 9628 catalog_manager.cc:4841] TS 79fab9fd748347559e7d7710682b465a (127.2.172.66:33489): tablet 81cccf5f6f29459087adc17c9076ef51 (table test-table [id=e9025fc3b17a4a0da345928d2c9c33da]) successfully deleted
I20260925 02:51:01.514722 10426 ts_tablet_manager.cc:1905] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:01.518530 10424 ts_tablet_manager.cc:1918] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:01.518747 10424 log.cc:1196] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/81cccf5f6f29459087adc17c9076ef51
I20260925 02:51:01.519312 10424 ts_tablet_manager.cc:1939] T 81cccf5f6f29459087adc17c9076ef51 P 080f175629c54c96b845358d2413c7f7: Deleting consensus metadata
I20260925 02:51:01.520406 10426 ts_tablet_manager.cc:1918] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:01.520637 10426 log.cc:1196] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/81cccf5f6f29459087adc17c9076ef51
I20260925 02:51:01.520730 9628 catalog_manager.cc:4841] TS 080f175629c54c96b845358d2413c7f7 (127.2.172.67:40383): tablet 81cccf5f6f29459087adc17c9076ef51 (table test-table [id=e9025fc3b17a4a0da345928d2c9c33da]) successfully deleted
I20260925 02:51:01.521193 10426 ts_tablet_manager.cc:1939] T 81cccf5f6f29459087adc17c9076ef51 P 30ab128117884673903a3c39d748b7a2: Deleting consensus metadata
I20260925 02:51:01.522483 9626 catalog_manager.cc:4841] TS 30ab128117884673903a3c39d748b7a2 (127.2.172.65:39097): tablet 81cccf5f6f29459087adc17c9076ef51 (table test-table [id=e9025fc3b17a4a0da345928d2c9c33da]) successfully deleted
I20260925 02:51:01.533977 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.505066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.2.172.65:39097)
0925 02:51:01.505268 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:01.505278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:01.519484 (+ 14206us) server_negotiation.cc:407] Connection header received
0925 02:51:01.519779 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:01.519785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:01.519885 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:01.520016 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:01.522089 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.523009 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.524381 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.524833 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.524949 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.527869 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:01.527886 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:01.527904 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:01.527940 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:01.530791 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:01.531656 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:01.531665 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:01.531667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:01.531823 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:01.532253 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:01.532257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:01.532261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:01.532653 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:01.532992 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:01.533347 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 02:51:01.533586 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:51:01.534858 10006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:01.601059 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.600631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39928)
0925 02:51:01.600809 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.600905 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 02:51:01.602046 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.601690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:36101 (local address 127.0.0.1:39936)
0925 02:51:01.601835 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:01.601917 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 02:51:01.670972 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 9818
I20260925 02:51:01.695533 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 10030
I20260925 02:51:01.710052 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.696423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.2.172.66:33489)
0925 02:51:01.697028 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:01.697033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:01.697047 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:51:01.697104 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:01.697107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:01.697173 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:01.697293 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:01.698724 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.699458 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.700878 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.701265 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.701406 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.703947 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:01.703964 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:01.703981 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:01.704028 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:01.706836 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:01.707673 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:01.707681 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:01.707684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:01.707818 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:01.708333 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:01.708338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:01.708342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:01.708729 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:01.709055 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:01.709460 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 02:51:01.709701 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":267,"threads_started":1}
I20260925 02:51:01.710938 10154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:01.828508 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 10030
I20260925 02:51:01.847186 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 10172
I20260925 02:51:01.861371 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:01.847996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.2.172.67:40383)
0925 02:51:01.848500 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:01.848505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:01.848524 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:51:01.848585 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:01.848590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:01.848652 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:01.848760 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:01.850162 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.850934 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.852197 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:01.852592 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:01.852686 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:01.855387 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:01.855404 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:01.855422 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:01.855455 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:01.858416 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:01.859134 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:01.859142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:01.859144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:01.859292 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:01.859846 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:01.859855 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:01.859857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:01.860245 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:01.860500 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:01.860787 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 02:51:01.860968 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":294,"threads_started":1}
I20260925 02:51:01.862073 10296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:01.958719 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 10172
I20260925 02:51:01.978152 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 9504
I20260925 02:51:01.979331 9672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:02.107465 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 9504
W20260925 02:51:02.129971 9816 connection.cc:535] server connection from 127.0.0.1:52864 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:51:02.130366 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 9605
I20260925 02:51:02.131654 9861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:02.248739 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 9605
2026-09-25T02:51:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10469 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:51:02 dist-test-slave-g1xh krb5kdc[10437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:51:02 dist-test-slave-g1xh krb5kdc[10437](info): set up 2 sockets
Sep 25 02:51:02 dist-test-slave-g1xh krb5kdc[10437](info): commencing operation
krb5kdc: starting...
W20260925 02:51:03.432754 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.007s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:51:03 dist-test-slave-g1xh krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:51:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.126.keytab.
Entry for principal oryx/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.126.keytab.
I20260925 02:51:03.686611 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:39009
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.126.keytab
--principal=oryx/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:39009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:04.027662 10450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:04.027980 10450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:04.028074 10450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:04.035826 10450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:04.035933 10450 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:51:04.035975 10450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:04.036048 10450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:04.036101 10450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:04.036160 10450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:04.046430 10450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:39009
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.126.keytab
--principal=oryx/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:39009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:04.047791 10450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:04.049136 10450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:04.062132 10455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:04.062234 10458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:04.062373 10456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:04.062971 10450 server_base.cc:1034] running on GCE node
I20260925 02:51:04.063874 10450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:04.065289 10450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:04.066555 10450 hybrid_clock.cc:648] HybridClock initialized: now 1790304664066487 us; error 40 us; skew 500 ppm
I20260925 02:51:04.066928 10450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:04 dist-test-slave-g1xh krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304664, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:04.070158 10450 init.cc:377] Logged in from keytab as oryx/127.2.172.126@KRBTEST.COM (short username oryx)
I20260925 02:51:04.071712 10450 webserver.cc:460] Webserver started at http://127.2.172.126:34117/ using document root <none> and password file <none>
I20260925 02:51:04.072419 10450 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:04.072554 10450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:04.072906 10450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:04.076113 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "82e89f6efe2e4236b72b353319c59a5f"
format_stamp: "Formatted at 2026-09-25 02:51:04 on dist-test-slave-g1xh"
server_key: "7c97247e39953678e75f994e16a849d8"
server_key_iv: "19f6291fc2e5944bb76cc3a6a8bf2266"
server_key_version: "encryptionkey@0"
I20260925 02:51:04.076934 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "82e89f6efe2e4236b72b353319c59a5f"
format_stamp: "Formatted at 2026-09-25 02:51:04 on dist-test-slave-g1xh"
server_key: "7c97247e39953678e75f994e16a849d8"
server_key_iv: "19f6291fc2e5944bb76cc3a6a8bf2266"
server_key_version: "encryptionkey@0"
I20260925 02:51:04.082557 10450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:51:04.086525 10465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:04.088294 10450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:51:04.088515 10450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "82e89f6efe2e4236b72b353319c59a5f"
format_stamp: "Formatted at 2026-09-25 02:51:04 on dist-test-slave-g1xh"
server_key: "7c97247e39953678e75f994e16a849d8"
server_key_iv: "19f6291fc2e5944bb76cc3a6a8bf2266"
server_key_version: "encryptionkey@0"
I20260925 02:51:04.088762 10450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:04.110056 10450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:04.114259 10450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:04.114544 10450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:04.137811 10450 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:39009
I20260925 02:51:04.137851 10527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:39009 every 8 connection(s)
I20260925 02:51:04.139230 10450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:51:04.140045 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 10450
I20260925 02:51:04.140386 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:51:04.141233 2737 external_mini_cluster.cc:1428] Setting key 56bd0e5413bf1c52cd75b3643c8263f2
I20260925 02:51:04.143893 10528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:51:04 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:51:04.157521 10528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f: Bootstrap starting.
I20260925 02:51:04.160741 10528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:04.161782 10528 log.cc:824] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:04.164763 10528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f: No bootstrap required, opened a new log
I20260925 02:51:04.164961 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:04.143459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.2.172.126:39009)
0925 02:51:04.144426 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:04.144438 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:04.144480 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:51:04.146250 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:04.146272 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:04.146675 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:04.147019 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:04.148940 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:04.150372 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:04.152140 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:04.152787 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:04.152924 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:04.157713 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:04.157728 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:04.157743 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:04.157785 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:04.160922 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:04.161639 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:04.161647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:04.161653 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:04.161814 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:04.162217 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:04.162225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:04.162227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:04.162757 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:04.163117 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:04.163520 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 02:51:04.163965 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":443,"threads_started":1}
I20260925 02:51:04.172762 10528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e89f6efe2e4236b72b353319c59a5f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39009 } }
I20260925 02:51:04.173205 10528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:04.173343 10528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e89f6efe2e4236b72b353319c59a5f, State: Initialized, Role: FOLLOWER
I20260925 02:51:04.173923 10528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e89f6efe2e4236b72b353319c59a5f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39009 } }
I20260925 02:51:04.174130 10528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:51:04.174221 10528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:51:04.174345 10528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:04.176517 10528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e89f6efe2e4236b72b353319c59a5f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39009 } }
I20260925 02:51:04.176991 10528 leader_election.cc:302] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82e89f6efe2e4236b72b353319c59a5f; no voters:
I20260925 02:51:04.177528 10528 leader_election.cc:288] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:51:04.177686 10533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:04.178519 10533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [term 1 LEADER]: Becoming Leader. State: Replica: 82e89f6efe2e4236b72b353319c59a5f, State: Running, Role: LEADER
I20260925 02:51:04.179246 10533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e89f6efe2e4236b72b353319c59a5f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39009 } }
I20260925 02:51:04.179913 10528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:51:04.182596 10534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82e89f6efe2e4236b72b353319c59a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e89f6efe2e4236b72b353319c59a5f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39009 } } }
I20260925 02:51:04.182582 10535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e89f6efe2e4236b72b353319c59a5f. Latest consensus state: current_term: 1 leader_uuid: "82e89f6efe2e4236b72b353319c59a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e89f6efe2e4236b72b353319c59a5f" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39009 } } }
I20260925 02:51:04.183035 10534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:04.183046 10535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:04.184525 10540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:04.190079 10540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:04.199321 10540 catalog_manager.cc:1347] Generated new cluster ID: 1f7d90141809408e80e7de86f20a684c
I20260925 02:51:04.199446 10540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:04.215675 10540 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:51:04.217732 10540 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:04.236938 10540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f: Generated new TSK 0
I20260925 02:51:04.237910 10540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab.
Entry for principal oryx/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab.
I20260925 02:51:04.303776 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:39009
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab
--principal=oryx/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:04.653204 10554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:04.653512 10554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:04.653609 10554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:04.661084 10554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:04.661212 10554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:04.661309 10554 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:51:04.661394 10554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:51:04.671900 10554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab
--principal=oryx/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.172.126:39009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:04.673200 10554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:04.674490 10554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:04.687965 10562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:04.688190 10560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:04.688427 10559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:04.688997 10554 server_base.cc:1034] running on GCE node
I20260925 02:51:04.689618 10554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:04.690560 10554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:04.691803 10554 hybrid_clock.cc:648] HybridClock initialized: now 1790304664691724 us; error 56 us; skew 500 ppm
I20260925 02:51:04.692106 10554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:04 dist-test-slave-g1xh krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304664, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:04.695432 10554 init.cc:377] Logged in from keytab as oryx/127.2.172.65@KRBTEST.COM (short username oryx)
I20260925 02:51:04.696951 10554 webserver.cc:460] Webserver started at http://127.2.172.65:33911/ using document root <none> and password file <none>
I20260925 02:51:04.697647 10554 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:04.697780 10554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:04.698099 10554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:04.701071 10554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "448d46b354074cb887780c697d77f80e"
format_stamp: "Formatted at 2026-09-25 02:51:04 on dist-test-slave-g1xh"
server_key: "8af494951d499bdee52b65409d4dde6c"
server_key_iv: "4110e8cc08cde7df4d5243320d6ceb5c"
server_key_version: "encryptionkey@0"
I20260925 02:51:04.701886 10554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "448d46b354074cb887780c697d77f80e"
format_stamp: "Formatted at 2026-09-25 02:51:04 on dist-test-slave-g1xh"
server_key: "8af494951d499bdee52b65409d4dde6c"
server_key_iv: "4110e8cc08cde7df4d5243320d6ceb5c"
server_key_version: "encryptionkey@0"
I20260925 02:51:04.707297 10554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 02:51:04.711218 10569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:04.712790 10554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:51:04.712984 10554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "448d46b354074cb887780c697d77f80e"
format_stamp: "Formatted at 2026-09-25 02:51:04 on dist-test-slave-g1xh"
server_key: "8af494951d499bdee52b65409d4dde6c"
server_key_iv: "4110e8cc08cde7df4d5243320d6ceb5c"
server_key_version: "encryptionkey@0"
I20260925 02:51:04.713253 10554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:04.732630 10554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:04.737681 10554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:04.737982 10554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:04.741182 10554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:04.741324 10554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:04.741550 10554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:04.741680 10554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:51:04 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304664, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:51:04.767959 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:04.741562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:41529 (local address 127.2.172.126:39009)
0925 02:51:04.742067 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:04.742072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:04.743609 (+ 1537us) server_negotiation.cc:407] Connection header received
0925 02:51:04.745728 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:04.745734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:04.745837 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:04.745972 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:04.748544 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:04.749890 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:04.752714 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:04.753501 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:04.753678 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:04.758816 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:04.758826 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:04.758834 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:04.758903 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:04.763272 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:04.764459 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:04.764465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:04.764469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:04.764654 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:04.765431 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:04.765439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:04.765444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:04.765792 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:04.765969 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:04.767253 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0925 02:51:04.767514 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":305,"threads_started":1}
I20260925 02:51:04.769928 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:04.742377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39009 (local address 127.2.172.65:41529)
0925 02:51:04.743381 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:04.743434 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:04.745398 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:04.746318 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:04.746336 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:04.747247 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:04.748277 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:04.748321 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:04.750148 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:04.750155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:04.752541 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:04.752557 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:04.753824 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:04.753832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:04.754058 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:04.754902 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:04.754930 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:04.758448 (+ 3518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:04.763494 (+ 5046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:04.763510 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:04.763537 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:04.764288 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:04.764812 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:04.764824 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:04.764831 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:04.765264 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:04.765965 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:04.765976 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:04.766669 (+ 693us) client_negotiation.cc:769] Sending connection context
0925 02:51:04.767087 (+ 418us) client_negotiation.cc:240] Negotiation successful
0925 02:51:04.767856 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":475,"threads_started":1}
I20260925 02:51:04.791343 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:04.776155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39009 (local address 127.2.172.65:57849)
0925 02:51:04.776840 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:04.776856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:04.776974 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:04.777423 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:04.777430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:04.777910 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:04.778200 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:04.778209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:04.780078 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:04.780085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:04.782011 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:04.782024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:04.782857 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:04.782875 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:04.783030 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:04.783725 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:04.783744 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:04.785068 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:04.788466 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:04.788471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:04.788475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:04.789087 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:04.789504 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:04.789508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:04.789512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:04.789657 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:04.790211 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:04.790215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:04.790457 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 02:51:04.790679 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 02:51:04.791004 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":361,"threads_started":1}
I20260925 02:51:04.791738 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:04.776084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:57849 (local address 127.2.172.126:39009)
0925 02:51:04.776354 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:04.776363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:04.776941 (+ 578us) server_negotiation.cc:407] Connection header received
0925 02:51:04.777099 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:04.777108 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:04.777216 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:04.777387 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:04.778357 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:04.779909 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:04.782171 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:04.782609 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:04.782755 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:04.785274 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:04.785279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:04.785283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:04.785314 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:04.788304 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:04.789216 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:04.789231 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:04.789235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:04.789375 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:04.789793 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:04.789801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:04.789806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:04.790085 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:04.790253 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:04.790830 (+ 577us) server_negotiation.cc:299] Negotiation successful
0925 02:51:04.791187 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:51:04.802228 10554 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:34873
I20260925 02:51:04.802260 10689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:34873 every 8 connection(s)
I20260925 02:51:04.803532 10554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:51:04.805928 10690 heartbeater.cc:347] Connected to a master server at 127.2.172.126:39009
I20260925 02:51:04.806152 10690 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:04.806854 10690 heartbeater.cc:511] Master 127.2.172.126:39009 requested a full tablet report, sending...
I20260925 02:51:04.808470 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 10554
I20260925 02:51:04.808715 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:51:04.808992 10482 ts_manager.cc:194] Registered new tserver with Master: 448d46b354074cb887780c697d77f80e (127.2.172.65:34873)
I20260925 02:51:04.809468 2737 external_mini_cluster.cc:1428] Setting key a0debebf3763b1f4cf014f6ab767f446
I20260925 02:51:04.810735 10482 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:41529
WARNING: no policy specified for oryx/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab.
Entry for principal oryx/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab.
I20260925 02:51:04.867149 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:39009
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab
--principal=oryx/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:05.216557 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:05.216837 10694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:05.216975 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:05.224449 10694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:05.224566 10694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:05.224650 10694 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:51:05.224732 10694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:51:05.234697 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab
--principal=oryx/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.172.126:39009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:05.236048 10694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:05.237383 10694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:05.251588 10702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:05.251704 10699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:05.251704 10700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:05.252800 10694 server_base.cc:1034] running on GCE node
I20260925 02:51:05.253463 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:05.254475 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:05.255729 10694 hybrid_clock.cc:648] HybridClock initialized: now 1790304665255644 us; error 30 us; skew 500 ppm
I20260925 02:51:05.256096 10694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:05 dist-test-slave-g1xh krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304665, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:05.260008 10694 init.cc:377] Logged in from keytab as oryx/127.2.172.66@KRBTEST.COM (short username oryx)
I20260925 02:51:05.261798 10694 webserver.cc:460] Webserver started at http://127.2.172.66:46319/ using document root <none> and password file <none>
I20260925 02:51:05.262598 10694 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:05.262758 10694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:05.263129 10694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:05.266487 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "011f95d37c2c453b937aad8b44cbc700"
format_stamp: "Formatted at 2026-09-25 02:51:05 on dist-test-slave-g1xh"
server_key: "296e03f882cf028546a2af80492100f9"
server_key_iv: "12034180af57a20d4688a55f533b4621"
server_key_version: "encryptionkey@0"
I20260925 02:51:05.267417 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "011f95d37c2c453b937aad8b44cbc700"
format_stamp: "Formatted at 2026-09-25 02:51:05 on dist-test-slave-g1xh"
server_key: "296e03f882cf028546a2af80492100f9"
server_key_iv: "12034180af57a20d4688a55f533b4621"
server_key_version: "encryptionkey@0"
I20260925 02:51:05.273538 10694 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:51:05.277709 10709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:05.279325 10694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:51:05.279625 10694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "011f95d37c2c453b937aad8b44cbc700"
format_stamp: "Formatted at 2026-09-25 02:51:05 on dist-test-slave-g1xh"
server_key: "296e03f882cf028546a2af80492100f9"
server_key_iv: "12034180af57a20d4688a55f533b4621"
server_key_version: "encryptionkey@0"
I20260925 02:51:05.279971 10694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:05.306289 10694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:05.310853 10694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:05.311177 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:05.313988 10694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:05.314121 10694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:05.314278 10694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:05.314349 10694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:51:05 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304665, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:51:05.342389 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:05.314796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:53949 (local address 127.2.172.126:39009)
0925 02:51:05.316633 (+ 1837us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:05.316639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:05.317158 (+ 519us) server_negotiation.cc:407] Connection header received
0925 02:51:05.319778 (+ 2620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:05.319786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:05.319892 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:05.320025 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:05.322822 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.324160 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.327521 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.328329 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.328487 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:05.334012 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:05.334020 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:05.334026 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:05.334071 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:05.338185 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:05.339382 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:05.339390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:05.339396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:05.339613 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:05.340098 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:05.340106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:05.340109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:05.340412 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:05.340596 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:05.341744 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0925 02:51:05.341984 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1705,"thread_start_us":1594,"threads_started":1}
I20260925 02:51:05.343606 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:05.315578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39009 (local address 127.2.172.66:53949)
0925 02:51:05.316972 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:05.317027 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:05.319433 (+ 2406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:05.320373 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:05.320390 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:05.321319 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:05.322581 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:05.322635 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.324361 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.324370 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:05.327327 (+ 2957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:05.327349 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.328652 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.328660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:05.329028 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:05.329977 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:05.330020 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:05.333710 (+ 3690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:05.338388 (+ 4678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:05.338398 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:05.338424 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:05.339242 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:05.339744 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:05.339750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:05.339754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:05.339961 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:05.340712 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:05.340723 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:05.341249 (+ 526us) client_negotiation.cc:769] Sending connection context
0925 02:51:05.341648 (+ 399us) client_negotiation.cc:240] Negotiation successful
0925 02:51:05.342265 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"thread_start_us":409,"threads_started":1}
I20260925 02:51:05.358042 10694 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:35319
I20260925 02:51:05.358144 10832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:35319 every 8 connection(s)
I20260925 02:51:05.359402 10694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:51:05.361865 10833 heartbeater.cc:347] Connected to a master server at 127.2.172.126:39009
I20260925 02:51:05.362208 10833 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:05.362923 10833 heartbeater.cc:511] Master 127.2.172.126:39009 requested a full tablet report, sending...
I20260925 02:51:05.363595 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 10694
I20260925 02:51:05.363831 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:51:05.364394 2737 external_mini_cluster.cc:1428] Setting key 034429d2a8e528af6c8885aa630b2ad3
I20260925 02:51:05.364650 10482 ts_manager.cc:194] Registered new tserver with Master: 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319)
I20260925 02:51:05.365681 10482 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:53949
I20260925 02:51:05.370433 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:05.354741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:33183 (local address 127.2.172.126:39009)
0925 02:51:05.354951 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:05.354958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:05.355791 (+ 833us) server_negotiation.cc:407] Connection header received
0925 02:51:05.355933 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:05.355942 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:05.356041 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:05.356150 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:05.357318 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.358466 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.359915 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.360610 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.360754 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:05.363172 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:05.363179 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:05.363183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:05.363221 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:05.366853 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:05.367746 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:05.367753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:05.367756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:05.367912 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:05.368419 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:05.368424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:05.368427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:05.368867 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:05.369523 (+ 656us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:05.369992 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 02:51:05.370200 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 02:51:05.370751 10827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:05.354905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39009 (local address 127.2.172.66:33183)
0925 02:51:05.355670 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:05.355688 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:05.355819 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:05.356206 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:05.356209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:05.356634 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:05.357116 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:05.357130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.358644 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.358648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:05.359758 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:05.359770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.361053 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.361059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:05.361259 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:05.361979 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:05.361996 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:05.362923 (+ 927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:05.367001 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:05.367008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:05.367014 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:05.367611 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:05.368049 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:05.368057 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:05.368063 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:05.368245 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:05.369384 (+ 1139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:05.369392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:05.369676 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 02:51:05.369830 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 02:51:05.370197 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":449,"threads_started":1}
WARNING: no policy specified for oryx/127.2.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab.
Entry for principal oryx/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab.
I20260925 02:51:05.418998 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:39009
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab
--principal=oryx/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:05.757419 10837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:05.757758 10837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:05.757941 10837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:05.765496 10837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:05.765661 10837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:05.765825 10837 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:51:05.765974 10837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:51:05.776132 10837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab
--principal=oryx/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.172.126:39009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:05.777532 10837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:05.778896 10837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:05.792771 10843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:05.792850 10845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:05.792806 10842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:05.793943 10837 server_base.cc:1034] running on GCE node
I20260925 02:51:05.794643 10837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:05.795715 10837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:05.797003 10837 hybrid_clock.cc:648] HybridClock initialized: now 1790304665796920 us; error 49 us; skew 500 ppm
I20260925 02:51:05.797375 10837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:05 dist-test-slave-g1xh krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304665, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:05.800822 10837 init.cc:377] Logged in from keytab as oryx/127.2.172.67@KRBTEST.COM (short username oryx)
I20260925 02:51:05.802572 10837 webserver.cc:460] Webserver started at http://127.2.172.67:46789/ using document root <none> and password file <none>
I20260925 02:51:05.803364 10837 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:05.803514 10837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:05.803913 10837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:05.807217 10837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "e41f43c3af1a4e47812247f3dad6704f"
format_stamp: "Formatted at 2026-09-25 02:51:05 on dist-test-slave-g1xh"
server_key: "ee384d3a14c07d6c91004cfb7ea53bca"
server_key_iv: "94a47702f60ec02e92027468b708af32"
server_key_version: "encryptionkey@0"
I20260925 02:51:05.808197 10837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "e41f43c3af1a4e47812247f3dad6704f"
format_stamp: "Formatted at 2026-09-25 02:51:05 on dist-test-slave-g1xh"
server_key: "ee384d3a14c07d6c91004cfb7ea53bca"
server_key_iv: "94a47702f60ec02e92027468b708af32"
server_key_version: "encryptionkey@0"
I20260925 02:51:05.814119 10837 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 02:51:05.814605 10690 heartbeater.cc:503] Master 127.2.172.126:39009 was elected leader, sending a full tablet report...
I20260925 02:51:05.818749 10852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:05.820529 10837 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:51:05.820744 10837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "e41f43c3af1a4e47812247f3dad6704f"
format_stamp: "Formatted at 2026-09-25 02:51:05 on dist-test-slave-g1xh"
server_key: "ee384d3a14c07d6c91004cfb7ea53bca"
server_key_iv: "94a47702f60ec02e92027468b708af32"
server_key_version: "encryptionkey@0"
I20260925 02:51:05.821015 10837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:05.844939 10837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:05.849467 10837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:05.849762 10837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:05.852717 10837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:05.852829 10837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:05.852985 10837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:05.853075 10837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:51:05 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304665, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:51:05.876992 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:05.853405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:47107 (local address 127.2.172.126:39009)
0925 02:51:05.853629 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:05.853637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:05.855027 (+ 1390us) server_negotiation.cc:407] Connection header received
0925 02:51:05.857160 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:05.857166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:05.857262 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:05.857379 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:05.859970 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.861056 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.863715 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.864337 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.864479 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:05.869917 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:05.869924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:05.869929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:05.869968 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:05.872817 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:05.874027 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:05.874032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:05.874034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:05.874175 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:05.874793 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:05.874798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:05.874801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:05.875177 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:05.875351 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:05.876558 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0925 02:51:05.876727 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:51:05.879019 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:05.854028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39009 (local address 127.2.172.67:47107)
0925 02:51:05.854845 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:05.854894 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:05.856864 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:05.857631 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:05.857640 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:05.858407 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:05.859783 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:05.859815 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.861289 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.861292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:05.863503 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:05.863526 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.864607 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.864616 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:05.864968 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:05.865823 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:05.865855 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:05.869575 (+ 3720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:05.872968 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:05.872981 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:05.873006 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:05.873786 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:05.874301 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:05.874309 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:05.874316 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:05.874623 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:05.875329 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:05.875342 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:05.875999 (+ 657us) client_negotiation.cc:769] Sending connection context
0925 02:51:05.876412 (+ 413us) client_negotiation.cc:240] Negotiation successful
0925 02:51:05.877139 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":354,"threads_started":1}
I20260925 02:51:05.903470 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:05.885607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39009 (local address 127.2.172.67:33745)
0925 02:51:05.886440 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:05.886472 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:05.886656 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:05.887006 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:05.887014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:05.887809 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:05.888345 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:05.888361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.889461 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.889469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:05.891540 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:05.891588 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.892612 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.892621 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:05.892842 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:05.893525 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:05.893578 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:05.895175 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:05.899794 (+ 4619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:05.899800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:05.899806 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:05.900478 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:05.901005 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:05.901013 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:05.901019 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:05.901239 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:05.901988 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:05.901993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:05.902265 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 02:51:05.902460 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 02:51:05.902869 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":427,"threads_started":1}
I20260925 02:51:05.904259 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:05.885694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:33745 (local address 127.2.172.126:39009)
0925 02:51:05.885948 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:05.885958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:05.886558 (+ 600us) server_negotiation.cc:407] Connection header received
0925 02:51:05.886787 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:05.886791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:05.886853 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:05.886935 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:05.888549 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.889316 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.891780 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.892298 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.892459 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:05.895439 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:05.895444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:05.895448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:05.895482 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:05.899485 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:05.900616 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:05.900626 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:05.900633 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:05.900846 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:05.901400 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:05.901406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:05.901409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:05.901852 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:05.902026 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:05.903598 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0925 02:51:05.903867 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:51:05.915804 10837 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:40889
I20260925 02:51:05.915836 10972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:40889 every 8 connection(s)
I20260925 02:51:05.917182 10837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:51:05.919492 10973 heartbeater.cc:347] Connected to a master server at 127.2.172.126:39009
I20260925 02:51:05.919752 10973 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:05.920375 10973 heartbeater.cc:511] Master 127.2.172.126:39009 requested a full tablet report, sending...
I20260925 02:51:05.921767 10481 ts_manager.cc:194] Registered new tserver with Master: e41f43c3af1a4e47812247f3dad6704f (127.2.172.67:40889)
I20260925 02:51:05.922709 10481 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:47107
I20260925 02:51:05.923671 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 10837
I20260925 02:51:05.923903 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:51:05.924422 2737 external_mini_cluster.cc:1428] Setting key c41267103eea5746bb2a66d1548f11e0
I20260925 02:51:05.927527 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:51:05 dist-test-slave-g1xh krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304665, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:51:05 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:05 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:06.370350 10833 heartbeater.cc:503] Master 127.2.172.126:39009 was elected leader, sending a full tablet report...
I20260925 02:51:06.925812 10973 heartbeater.cc:503] Master 127.2.172.126:39009 was elected leader, sending a full tablet report...
W20260925 02:51:07.089413 10981 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.112s user 0.002s sys 0.016s
W20260925 02:51:07.090754 10719 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:05.971599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.2.172.126:39009)
0925 02:51:05.971852 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:05.971859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:05.972059 (+ 200us) server_negotiation.cc:407] Connection header received
0925 02:51:05.972294 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:05.972300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:05.972378 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:05.972489 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:05.973952 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.974712 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.976114 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:05.976533 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:05.976694 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.090409 (+1113715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43276: BlockingRecv error: recv got EOF from 127.0.0.1:43276 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 02:51:07 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304665, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:51:07.117097 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.095096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.2.172.126:39009)
0925 02:51:07.095407 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.095413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.095665 (+ 252us) server_negotiation.cc:407] Connection header received
0925 02:51:07.095892 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.095902 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.095967 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.096073 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:07.097321 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.098063 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.099319 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.099834 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.099977 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.111780 (+ 11803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:07.111787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:07.111791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:07.111836 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:07.114566 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:07.115288 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:07.115292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:07.115294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:07.115417 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:07.115792 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:07.115795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:07.115797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:07.116073 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:07.116326 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:07.116681 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 02:51:07.116855 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260925 02:51:07.121649 10481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:51:07.129160 10481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:51:07.167642 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.142280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:34873 (local address 127.0.0.1:45646)
0925 02:51:07.143186 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:07.143234 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:07.143390 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:07.157208 (+ 13818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:07.157221 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:07.157285 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:07.158015 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.158033 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.160769 (+ 2736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.160775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.163781 (+ 3006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.163804 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.166325 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.166334 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.166639 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.166710 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 02:51:07.166862 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 02:51:07.167123 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":527,"threads_started":1}
I20260925 02:51:07.169122 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.142776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.2.172.65:34873)
0925 02:51:07.156191 (+ 13415us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.156206 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.156258 (+ 52us) server_negotiation.cc:407] Connection header received
0925 02:51:07.156411 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.156423 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.156819 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.157131 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:07.158699 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.160603 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.164046 (+ 3443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.165704 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.165865 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.166137 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:07.167274 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0925 02:51:07.167541 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13177,"thread_start_us":12982,"threads_started":1}
I20260925 02:51:07.173236 10620 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3d470219144456b2f0c05c515849fb (DEFAULT_TABLE table=test-table [id=da2ed9b463584f3ca937ab75ed642d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:07.174763 10620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3d470219144456b2f0c05c515849fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:07.176132 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.144287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:35319 (local address 127.0.0.1:60942)
0925 02:51:07.167979 (+ 23692us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:07.167999 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:07.168100 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:07.169001 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:07.169005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:07.169034 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:07.169204 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.169215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.171127 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.171134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.172985 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.173001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.175019 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.175026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.175200 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.175234 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:51:07.175347 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:51:07.175538 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23545,"thread_start_us":25043,"threads_started":1}
I20260925 02:51:07.176263 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.144920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.2.172.66:35319)
0925 02:51:07.168253 (+ 23333us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.168266 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.168304 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:51:07.168435 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.168447 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.168768 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.169587 (+ 819us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:07.169821 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.170961 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.173193 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.174730 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.174877 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.175091 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:07.175774 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 02:51:07.175953 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23169,"thread_start_us":23022,"threads_started":1}
I20260925 02:51:07.176518 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.142722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:40889 (local address 127.0.0.1:46234)
0925 02:51:07.158366 (+ 15644us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:07.158397 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:07.158521 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:07.169503 (+ 10982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:07.169508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:07.169536 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:07.169860 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.169869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.171819 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.171824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.173912 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.173928 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.175785 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.175791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.175962 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.175993 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:51:07.176060 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 02:51:07.176197 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15170,"thread_start_us":14856,"threads_started":1}
I20260925 02:51:07.177318 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.143655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.2.172.67:40889)
0925 02:51:07.168873 (+ 25218us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.168881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.168909 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:51:07.169018 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.169026 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.169254 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.170031 (+ 777us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:07.170177 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.171667 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.174113 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.175520 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.176548 (+ 1028us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.176773 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:07.176854 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 02:51:07.177025 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25069,"thread_start_us":24942,"threads_started":1}
I20260925 02:51:07.180305 10903 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3d470219144456b2f0c05c515849fb (DEFAULT_TABLE table=test-table [id=da2ed9b463584f3ca937ab75ed642d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:07.180532 10762 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3d470219144456b2f0c05c515849fb (DEFAULT_TABLE table=test-table [id=da2ed9b463584f3ca937ab75ed642d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:07.181845 10903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3d470219144456b2f0c05c515849fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:07.182070 10762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3d470219144456b2f0c05c515849fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:07.192375 10998 tablet_bootstrap.cc:492] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e: Bootstrap starting.
I20260925 02:51:07.194325 10999 tablet_bootstrap.cc:492] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f: Bootstrap starting.
I20260925 02:51:07.196460 10998 tablet_bootstrap.cc:654] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:07.197952 10999 tablet_bootstrap.cc:654] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:07.198170 11000 tablet_bootstrap.cc:492] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700: Bootstrap starting.
I20260925 02:51:07.198272 10998 log.cc:824] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:07.199128 10999 log.cc:824] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:07.201381 11000 tablet_bootstrap.cc:654] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:07.201470 10998 tablet_bootstrap.cc:492] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e: No bootstrap required, opened a new log
I20260925 02:51:07.201745 10998 ts_tablet_manager.cc:1397] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 02:51:07.202111 10999 tablet_bootstrap.cc:492] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f: No bootstrap required, opened a new log
I20260925 02:51:07.202378 10999 ts_tablet_manager.cc:1397] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 02:51:07.202421 11000 log.cc:824] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:07.206750 11000 tablet_bootstrap.cc:492] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700: No bootstrap required, opened a new log
I20260925 02:51:07.207054 11000 ts_tablet_manager.cc:1397] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260925 02:51:07.212823 10998 raft_consensus.cc:348] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } }
I20260925 02:51:07.213351 10998 raft_consensus.cc:374] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:07.213488 10998 raft_consensus.cc:729] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448d46b354074cb887780c697d77f80e, State: Initialized, Role: FOLLOWER
I20260925 02:51:07.213229 10999 raft_consensus.cc:348] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } }
I20260925 02:51:07.213644 10999 raft_consensus.cc:374] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:07.213869 10999 raft_consensus.cc:729] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41f43c3af1a4e47812247f3dad6704f, State: Initialized, Role: FOLLOWER
I20260925 02:51:07.214166 10998 consensus_queue.cc:260] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } }
I20260925 02:51:07.214439 10999 consensus_queue.cc:260] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } }
I20260925 02:51:07.215050 11000 raft_consensus.cc:348] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } }
I20260925 02:51:07.215447 11000 raft_consensus.cc:374] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:07.215546 11000 raft_consensus.cc:729] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011f95d37c2c453b937aad8b44cbc700, State: Initialized, Role: FOLLOWER
I20260925 02:51:07.216151 11000 consensus_queue.cc:260] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } }
I20260925 02:51:07.216552 10998 ts_tablet_manager.cc:1428] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260925 02:51:07.221454 11000 ts_tablet_manager.cc:1428] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260925 02:51:07.223174 10999 ts_tablet_manager.cc:1428] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f: Time spent starting tablet: real 0.021s user 0.005s sys 0.016s
W20260925 02:51:07.307211 10691 tablet.cc:2369] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:51:07.309520 11006 raft_consensus.cc:482] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:07.309805 11006 raft_consensus.cc:504] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } }
I20260925 02:51:07.311771 11006 leader_election.cc:288] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 448d46b354074cb887780c697d77f80e (127.2.172.65:34873), 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319)
I20260925 02:51:07.321750 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.312130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:58279 (local address 127.2.172.65:34873)
0925 02:51:07.312386 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.312395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.313441 (+ 1046us) server_negotiation.cc:407] Connection header received
0925 02:51:07.313852 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.313858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.313963 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.314075 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:07.314803 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.316000 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.319170 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.320533 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.320712 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.320791 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:07.321349 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 02:51:07.321487 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:51:07.321776 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.312353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:58135 (local address 127.2.172.66:35319)
0925 02:51:07.312643 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.312652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.313158 (+ 506us) server_negotiation.cc:407] Connection header received
0925 02:51:07.313517 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.313523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.313706 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.313844 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:07.314583 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.315866 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.319171 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.320533 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.320712 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.320791 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:07.321349 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 02:51:07.321486 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 02:51:07.321997 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.312113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:34873 (local address 127.2.172.67:58279)
0925 02:51:07.313340 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:07.313360 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:07.313592 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:07.314099 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:07.314104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:07.314137 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:07.314625 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.314636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.316166 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.316175 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.318941 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.318960 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.320890 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.320896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.321134 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.321192 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 02:51:07.321307 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:51:07.321497 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"mutex_wait_us":249,"thread_start_us":504,"threads_started":1}
I20260925 02:51:07.322019 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.312094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:35319 (local address 127.2.172.67:58135)
0925 02:51:07.313010 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:07.313036 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:07.313300 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:07.313847 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:07.313853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:07.313882 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:07.314398 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.314416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.316026 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.316030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.318941 (+ 2911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.318960 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.320865 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.320879 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.321139 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.321192 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 02:51:07.321307 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:51:07.321497 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":262,"thread_start_us":946,"threads_started":1}
I20260925 02:51:07.323333 10640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3d470219144456b2f0c05c515849fb" candidate_uuid: "e41f43c3af1a4e47812247f3dad6704f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448d46b354074cb887780c697d77f80e" is_pre_election: true
I20260925 02:51:07.323369 10782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3d470219144456b2f0c05c515849fb" candidate_uuid: "e41f43c3af1a4e47812247f3dad6704f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011f95d37c2c453b937aad8b44cbc700" is_pre_election: true
I20260925 02:51:07.324002 10782 raft_consensus.cc:2454] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41f43c3af1a4e47812247f3dad6704f in term 0.
I20260925 02:51:07.324002 10640 raft_consensus.cc:2454] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41f43c3af1a4e47812247f3dad6704f in term 0.
I20260925 02:51:07.324776 10853 leader_election.cc:302] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448d46b354074cb887780c697d77f80e, e41f43c3af1a4e47812247f3dad6704f; no voters:
I20260925 02:51:07.325394 11006 raft_consensus.cc:2781] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:07.325565 11006 raft_consensus.cc:482] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:07.325683 11006 raft_consensus.cc:3037] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:07.328019 11006 raft_consensus.cc:504] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } }
I20260925 02:51:07.328668 11006 leader_election.cc:288] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [CANDIDATE]: Term 1 election: Requested vote from peers 448d46b354074cb887780c697d77f80e (127.2.172.65:34873), 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319)
I20260925 02:51:07.329329 10640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3d470219144456b2f0c05c515849fb" candidate_uuid: "e41f43c3af1a4e47812247f3dad6704f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448d46b354074cb887780c697d77f80e"
I20260925 02:51:07.329337 10782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3d470219144456b2f0c05c515849fb" candidate_uuid: "e41f43c3af1a4e47812247f3dad6704f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011f95d37c2c453b937aad8b44cbc700"
I20260925 02:51:07.329568 10640 raft_consensus.cc:3037] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:07.329568 10782 raft_consensus.cc:3037] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:07.332904 10640 raft_consensus.cc:2454] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41f43c3af1a4e47812247f3dad6704f in term 1.
I20260925 02:51:07.332904 10782 raft_consensus.cc:2454] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41f43c3af1a4e47812247f3dad6704f in term 1.
I20260925 02:51:07.333425 10853 leader_election.cc:302] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 011f95d37c2c453b937aad8b44cbc700, 448d46b354074cb887780c697d77f80e, e41f43c3af1a4e47812247f3dad6704f; no voters:
I20260925 02:51:07.333809 11006 raft_consensus.cc:2781] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:07.334348 11006 raft_consensus.cc:686] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 1 LEADER]: Becoming Leader. State: Replica: e41f43c3af1a4e47812247f3dad6704f, State: Running, Role: LEADER
I20260925 02:51:07.334896 11006 consensus_queue.cc:237] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } }
I20260925 02:51:07.339751 10482 catalog_manager.cc:5495] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f reported cstate change: term changed from 0 to 1, leader changed from <none> to e41f43c3af1a4e47812247f3dad6704f (127.2.172.67). New cstate: current_term: 1 leader_uuid: "e41f43c3af1a4e47812247f3dad6704f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } health_report { overall_health: UNKNOWN } } }
W20260925 02:51:07.362628 10834 tablet.cc:2369] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:51:07.364262 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.357909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.2.172.67:40889)
0925 02:51:07.358113 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.358117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.358182 (+ 65us) server_negotiation.cc:407] Connection header received
0925 02:51:07.358447 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.358451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.358543 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.358662 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:07.359037 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.359831 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.361847 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.362244 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.362398 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.362843 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:07.363655 (+ 812us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:07.363772 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:07.363896 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 02:51:07.363962 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:51:07.373492 10782 raft_consensus.cc:1261] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Refusing update from remote peer e41f43c3af1a4e47812247f3dad6704f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:07.373492 10640 raft_consensus.cc:1261] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Refusing update from remote peer e41f43c3af1a4e47812247f3dad6704f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:07.374612 11006 consensus_queue.cc:1035] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [LEADER]: Connected to new peer: Peer: permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:07.374984 11010 consensus_queue.cc:1035] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [LEADER]: Connected to new peer: Peer: permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:07.401494 10482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:07.401785 10482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:07.405422 10482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f: Sending DeleteTablet for 3 replicas of tablet 8d3d470219144456b2f0c05c515849fb
I20260925 02:51:07.406581 10620 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3d470219144456b2f0c05c515849fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:07 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:45646
I20260925 02:51:07.406607 10762 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3d470219144456b2f0c05c515849fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:07 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:60942
I20260925 02:51:07.406627 10903 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3d470219144456b2f0c05c515849fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:07 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:46234
I20260925 02:51:07.409401 10480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:51:07.411757 10480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:51:07.422005 10619 tablet_service.cc:1467] Processing CreateTablet for tablet b94f0306b4d84517972d3621cb7382cf (DEFAULT_TABLE table=test-table [id=e83393dbe64e46e18d9604f1e0ca1bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:07.422676 10619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94f0306b4d84517972d3621cb7382cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:07.422523 10761 tablet_service.cc:1467] Processing CreateTablet for tablet b94f0306b4d84517972d3621cb7382cf (DEFAULT_TABLE table=test-table [id=e83393dbe64e46e18d9604f1e0ca1bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:07.423183 10761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94f0306b4d84517972d3621cb7382cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:07.428815 10902 tablet_service.cc:1467] Processing CreateTablet for tablet b94f0306b4d84517972d3621cb7382cf (DEFAULT_TABLE table=test-table [id=e83393dbe64e46e18d9604f1e0ca1bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:07.429575 10902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94f0306b4d84517972d3621cb7382cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:07.431847 10998 tablet_bootstrap.cc:492] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e: Bootstrap starting.
I20260925 02:51:07.434144 10998 tablet_bootstrap.cc:654] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:07.442545 10998 tablet_bootstrap.cc:492] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e: No bootstrap required, opened a new log
I20260925 02:51:07.442785 10998 ts_tablet_manager.cc:1397] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20260925 02:51:07.443748 10998 raft_consensus.cc:348] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:07.444057 10998 raft_consensus.cc:374] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:07.444228 10998 raft_consensus.cc:729] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448d46b354074cb887780c697d77f80e, State: Initialized, Role: FOLLOWER
I20260925 02:51:07.444589 10998 consensus_queue.cc:260] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:07.445917 10998 ts_tablet_manager.cc:1428] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:51:07.463315 11000 tablet_bootstrap.cc:492] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700: Bootstrap starting.
I20260925 02:51:07.465305 11000 tablet_bootstrap.cc:654] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:07.467783 10999 tablet_bootstrap.cc:492] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f: Bootstrap starting.
I20260925 02:51:07.470079 10999 tablet_bootstrap.cc:654] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:07.475584 11023 tablet_replica.cc:330] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700: stopping tablet replica
I20260925 02:51:07.476671 11024 tablet_replica.cc:330] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e: stopping tablet replica
I20260925 02:51:07.477108 11024 raft_consensus.cc:2229] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:07.477612 11024 raft_consensus.cc:2258] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:07.478227 11023 raft_consensus.cc:2229] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:07.478700 11023 raft_consensus.cc:2258] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:07.481860 11023 ts_tablet_manager.cc:1905] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:07.482782 11025 tablet_replica.cc:330] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f: stopping tablet replica
I20260925 02:51:07.483265 11025 raft_consensus.cc:2229] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:51:07.482172 11024 ts_tablet_manager.cc:1905] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:07.483577 11000 tablet_bootstrap.cc:492] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700: No bootstrap required, opened a new log
I20260925 02:51:07.483783 11000 ts_tablet_manager.cc:1397] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.005s
I20260925 02:51:07.485024 11000 raft_consensus.cc:348] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:07.485347 11000 raft_consensus.cc:374] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:07.485532 11000 raft_consensus.cc:729] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011f95d37c2c453b937aad8b44cbc700, State: Initialized, Role: FOLLOWER
I20260925 02:51:07.485702 11025 raft_consensus.cc:2258] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:07.485884 11000 consensus_queue.cc:260] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:07.487888 11025 ts_tablet_manager.cc:1905] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:07.489317 11000 ts_tablet_manager.cc:1428] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260925 02:51:07.490221 11023 ts_tablet_manager.cc:1918] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:07.490463 11023 log.cc:1196] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/8d3d470219144456b2f0c05c515849fb
I20260925 02:51:07.491112 11023 ts_tablet_manager.cc:1939] T 8d3d470219144456b2f0c05c515849fb P 011f95d37c2c453b937aad8b44cbc700: Deleting consensus metadata
I20260925 02:51:07.492995 10999 tablet_bootstrap.cc:492] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f: No bootstrap required, opened a new log
I20260925 02:51:07.493172 10999 ts_tablet_manager.cc:1397] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.009s
I20260925 02:51:07.493335 10469 catalog_manager.cc:4841] TS 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319): tablet 8d3d470219144456b2f0c05c515849fb (table test-table [id=da2ed9b463584f3ca937ab75ed642d92]) successfully deleted
I20260925 02:51:07.494165 11024 ts_tablet_manager.cc:1918] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:07.494040 10999 raft_consensus.cc:348] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:07.494279 10999 raft_consensus.cc:374] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:07.494359 11024 log.cc:1196] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/8d3d470219144456b2f0c05c515849fb
I20260925 02:51:07.494374 10999 raft_consensus.cc:729] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41f43c3af1a4e47812247f3dad6704f, State: Initialized, Role: FOLLOWER
I20260925 02:51:07.494697 10999 consensus_queue.cc:260] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:07.494907 11024 ts_tablet_manager.cc:1939] T 8d3d470219144456b2f0c05c515849fb P 448d46b354074cb887780c697d77f80e: Deleting consensus metadata
I20260925 02:51:07.495384 10999 ts_tablet_manager.cc:1428] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 02:51:07.496855 10466 catalog_manager.cc:4841] TS 448d46b354074cb887780c697d77f80e (127.2.172.65:34873): tablet 8d3d470219144456b2f0c05c515849fb (table test-table [id=da2ed9b463584f3ca937ab75ed642d92]) successfully deleted
I20260925 02:51:07.499408 11025 ts_tablet_manager.cc:1918] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:07.499632 11025 log.cc:1196] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/8d3d470219144456b2f0c05c515849fb
I20260925 02:51:07.500213 11025 ts_tablet_manager.cc:1939] T 8d3d470219144456b2f0c05c515849fb P e41f43c3af1a4e47812247f3dad6704f: Deleting consensus metadata
I20260925 02:51:07.502013 10467 catalog_manager.cc:4841] TS e41f43c3af1a4e47812247f3dad6704f (127.2.172.67:40889): tablet 8d3d470219144456b2f0c05c515849fb (table test-table [id=da2ed9b463584f3ca937ab75ed642d92]) successfully deleted
W20260925 02:51:07.671042 10974 tablet.cc:2369] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:51:07.701776 11007 raft_consensus.cc:482] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:07.702016 11007 raft_consensus.cc:504] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:07.703624 11007 leader_election.cc:288] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 448d46b354074cb887780c697d77f80e (127.2.172.65:34873), e41f43c3af1a4e47812247f3dad6704f (127.2.172.67:40889)
I20260925 02:51:07.712212 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.704022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:55955 (local address 127.2.172.67:40889)
0925 02:51:07.704218 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.704225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.705534 (+ 1309us) server_negotiation.cc:407] Connection header received
0925 02:51:07.705719 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.705725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.705813 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.705932 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:07.706642 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.707774 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.709591 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.710966 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.711154 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.711238 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:07.711698 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 02:51:07.711856 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 02:51:07.712199 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.704021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:48175 (local address 127.2.172.65:34873)
0925 02:51:07.704221 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.704227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.704925 (+ 698us) server_negotiation.cc:407] Connection header received
0925 02:51:07.705211 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.705217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.705301 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.705375 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:07.706262 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.707321 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.709946 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.711164 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.711336 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.711414 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:07.711817 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 02:51:07.711939 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:51:07.712649 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.704503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:40889 (local address 127.2.172.66:55955)
0925 02:51:07.705410 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:07.705428 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:07.705559 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:07.705970 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:07.705976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:07.706006 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:07.706450 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.706462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.707909 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.707916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.709423 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.709435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.711195 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.711201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.711492 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.711529 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:51:07.711670 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 02:51:07.711892 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":26,"thread_start_us":569,"threads_started":1}
I20260925 02:51:07.713110 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.704041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:34873 (local address 127.2.172.66:48175)
0925 02:51:07.704777 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:07.704803 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:07.704989 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:07.705590 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:07.705601 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:07.705649 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:07.706046 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.706061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.707473 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.707480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.709771 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.709789 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.711408 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.711414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.711617 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.711656 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:51:07.712526 (+ 870us) client_negotiation.cc:240] Negotiation successful
0925 02:51:07.712642 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":1,"thread_start_us":373,"threads_started":1}
I20260925 02:51:07.713723 10640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94f0306b4d84517972d3621cb7382cf" candidate_uuid: "011f95d37c2c453b937aad8b44cbc700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448d46b354074cb887780c697d77f80e" is_pre_election: true
I20260925 02:51:07.713670 10928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94f0306b4d84517972d3621cb7382cf" candidate_uuid: "011f95d37c2c453b937aad8b44cbc700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41f43c3af1a4e47812247f3dad6704f" is_pre_election: true
I20260925 02:51:07.714010 10640 raft_consensus.cc:2454] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 011f95d37c2c453b937aad8b44cbc700 in term 0.
I20260925 02:51:07.714143 10928 raft_consensus.cc:2454] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 011f95d37c2c453b937aad8b44cbc700 in term 0.
I20260925 02:51:07.714524 10711 leader_election.cc:302] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011f95d37c2c453b937aad8b44cbc700, 448d46b354074cb887780c697d77f80e; no voters:
I20260925 02:51:07.715224 11007 raft_consensus.cc:2781] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:07.715344 11007 raft_consensus.cc:482] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:07.715435 11007 raft_consensus.cc:3037] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:07.718685 11007 raft_consensus.cc:504] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:07.719455 11007 leader_election.cc:288] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [CANDIDATE]: Term 1 election: Requested vote from peers 448d46b354074cb887780c697d77f80e (127.2.172.65:34873), e41f43c3af1a4e47812247f3dad6704f (127.2.172.67:40889)
I20260925 02:51:07.720008 10928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94f0306b4d84517972d3621cb7382cf" candidate_uuid: "011f95d37c2c453b937aad8b44cbc700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41f43c3af1a4e47812247f3dad6704f"
I20260925 02:51:07.720027 10640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94f0306b4d84517972d3621cb7382cf" candidate_uuid: "011f95d37c2c453b937aad8b44cbc700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448d46b354074cb887780c697d77f80e"
I20260925 02:51:07.720258 10640 raft_consensus.cc:3037] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:07.720278 10928 raft_consensus.cc:3037] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:07.722668 10928 raft_consensus.cc:2454] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 011f95d37c2c453b937aad8b44cbc700 in term 1.
I20260925 02:51:07.723141 10712 leader_election.cc:302] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011f95d37c2c453b937aad8b44cbc700, e41f43c3af1a4e47812247f3dad6704f; no voters:
I20260925 02:51:07.723266 10640 raft_consensus.cc:2454] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 011f95d37c2c453b937aad8b44cbc700 in term 1.
I20260925 02:51:07.723500 11007 raft_consensus.cc:2781] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:07.724098 11007 raft_consensus.cc:686] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 1 LEADER]: Becoming Leader. State: Replica: 011f95d37c2c453b937aad8b44cbc700, State: Running, Role: LEADER
I20260925 02:51:07.724534 11007 consensus_queue.cc:237] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:07.728415 10480 catalog_manager.cc:5495] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 reported cstate change: term changed from 0 to 1, leader changed from <none> to 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66). New cstate: current_term: 1 leader_uuid: "011f95d37c2c453b937aad8b44cbc700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } health_report { overall_health: UNKNOWN } } }
Sep 25 02:51:07 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304664, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.126@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:51:07.813339 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.794771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39009 (local address 127.0.0.1:43298)
0925 02:51:07.795504 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:07.795527 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:07.795745 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:07.796162 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:07.796167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:07.797134 (+ 967us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:07.797445 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.797462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.798832 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.798840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.800497 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.800514 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.801332 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.801338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.801530 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.802457 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:07.802481 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:07.805594 (+ 3113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:07.809965 (+ 4371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:07.809984 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:07.809993 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:07.810646 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:07.811107 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:07.811112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:07.811116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:07.811300 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:07.812059 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:07.812069 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:07.812280 (+ 211us) client_negotiation.cc:769] Sending connection context
0925 02:51:07.812472 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 02:51:07.812822 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":362,"threads_started":1}
I20260925 02:51:07.813815 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.794784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.2.172.126:39009)
0925 02:51:07.795855 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.795860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.795881 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:51:07.795926 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.795930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.795984 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.796072 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:07.797688 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.798667 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.800708 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.801133 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.801241 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.805870 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:07.805879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:07.805884 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:07.805931 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:07.809789 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:07.810807 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:07.810814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:07.810818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:07.810969 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:07.811433 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:07.811439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:07.811443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:07.811890 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:07.812066 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:07.812601 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 02:51:07.812999 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":974,"thread_start_us":794,"threads_started":1}
I20260925 02:51:07.820173 10480 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:43298:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:51:07.821600 10480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:51:07.830065 10903 tablet_service.cc:1467] Processing CreateTablet for tablet 807515a54db44eef86511e8564d3d8b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5b90edf6264453794f677957339e407]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:51:07.829989 10762 tablet_service.cc:1467] Processing CreateTablet for tablet 807515a54db44eef86511e8564d3d8b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5b90edf6264453794f677957339e407]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:51:07.830612 10903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 807515a54db44eef86511e8564d3d8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:07.830405 10620 tablet_service.cc:1467] Processing CreateTablet for tablet 807515a54db44eef86511e8564d3d8b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5b90edf6264453794f677957339e407]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:51:07.830734 10762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 807515a54db44eef86511e8564d3d8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:07.830996 10620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 807515a54db44eef86511e8564d3d8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:07.838598 10999 tablet_bootstrap.cc:492] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: Bootstrap starting.
I20260925 02:51:07.838995 10998 tablet_bootstrap.cc:492] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e: Bootstrap starting.
I20260925 02:51:07.840469 11000 tablet_bootstrap.cc:492] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700: Bootstrap starting.
I20260925 02:51:07.840612 10999 tablet_bootstrap.cc:654] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:07.840947 10998 tablet_bootstrap.cc:654] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:07.842648 11000 tablet_bootstrap.cc:654] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:07.851646 11000 tablet_bootstrap.cc:492] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700: No bootstrap required, opened a new log
I20260925 02:51:07.851842 11000 ts_tablet_manager.cc:1397] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260925 02:51:07.851842 10998 tablet_bootstrap.cc:492] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e: No bootstrap required, opened a new log
I20260925 02:51:07.852043 10998 ts_tablet_manager.cc:1397] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260925 02:51:07.852586 11000 raft_consensus.cc:348] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } }
I20260925 02:51:07.852838 11000 raft_consensus.cc:374] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:07.852918 11000 raft_consensus.cc:729] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011f95d37c2c453b937aad8b44cbc700, State: Initialized, Role: FOLLOWER
I20260925 02:51:07.853055 10999 tablet_bootstrap.cc:492] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: No bootstrap required, opened a new log
I20260925 02:51:07.852893 10998 raft_consensus.cc:348] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } }
I20260925 02:51:07.853188 10998 raft_consensus.cc:374] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:07.853273 10999 ts_tablet_manager.cc:1397] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260925 02:51:07.853291 10998 raft_consensus.cc:729] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448d46b354074cb887780c697d77f80e, State: Initialized, Role: FOLLOWER
I20260925 02:51:07.853161 11000 consensus_queue.cc:260] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } }
I20260925 02:51:07.853629 10998 consensus_queue.cc:260] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } }
I20260925 02:51:07.853796 11000 ts_tablet_manager.cc:1428] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:51:07.854105 11007 tablet_replica.cc:439] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } }
I20260925 02:51:07.854127 11005 tablet_replica.cc:439] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } }
I20260925 02:51:07.854399 11007 tablet_replica.cc:442] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:51:07.854509 11005 tablet_replica.cc:442] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:51:07.855782 10998 ts_tablet_manager.cc:1428] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 02:51:07.854130 10999 raft_consensus.cc:348] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } }
I20260925 02:51:07.856389 10999 raft_consensus.cc:374] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:07.856544 10999 raft_consensus.cc:729] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41f43c3af1a4e47812247f3dad6704f, State: Initialized, Role: FOLLOWER
I20260925 02:51:07.856875 10999 consensus_queue.cc:260] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } }
I20260925 02:51:07.857432 11018 tablet_replica.cc:439] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } }
I20260925 02:51:07.857672 10999 ts_tablet_manager.cc:1428] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260925 02:51:07.857793 11018 tablet_replica.cc:442] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:51:07.894254 11018 raft_consensus.cc:482] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:07.894496 11018 raft_consensus.cc:504] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } }
I20260925 02:51:07.895242 11018 leader_election.cc:288] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319), 448d46b354074cb887780c697d77f80e (127.2.172.65:34873)
I20260925 02:51:07.895897 10640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "807515a54db44eef86511e8564d3d8b7" candidate_uuid: "e41f43c3af1a4e47812247f3dad6704f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448d46b354074cb887780c697d77f80e" is_pre_election: true
I20260925 02:51:07.895985 10782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "807515a54db44eef86511e8564d3d8b7" candidate_uuid: "e41f43c3af1a4e47812247f3dad6704f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011f95d37c2c453b937aad8b44cbc700" is_pre_election: true
I20260925 02:51:07.896188 10640 raft_consensus.cc:2454] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41f43c3af1a4e47812247f3dad6704f in term 0.
I20260925 02:51:07.896234 10782 raft_consensus.cc:2454] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41f43c3af1a4e47812247f3dad6704f in term 0.
I20260925 02:51:07.896739 10853 leader_election.cc:302] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448d46b354074cb887780c697d77f80e, e41f43c3af1a4e47812247f3dad6704f; no voters:
I20260925 02:51:07.897045 11018 raft_consensus.cc:2781] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:07.897256 11018 raft_consensus.cc:482] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:07.897356 11018 raft_consensus.cc:3037] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:07.900012 11018 raft_consensus.cc:504] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } }
I20260925 02:51:07.900677 11018 leader_election.cc:288] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [CANDIDATE]: Term 1 election: Requested vote from peers 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319), 448d46b354074cb887780c697d77f80e (127.2.172.65:34873)
I20260925 02:51:07.901221 10640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "807515a54db44eef86511e8564d3d8b7" candidate_uuid: "e41f43c3af1a4e47812247f3dad6704f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448d46b354074cb887780c697d77f80e"
I20260925 02:51:07.901252 10782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "807515a54db44eef86511e8564d3d8b7" candidate_uuid: "e41f43c3af1a4e47812247f3dad6704f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011f95d37c2c453b937aad8b44cbc700"
I20260925 02:51:07.901456 10782 raft_consensus.cc:3037] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:07.901479 10640 raft_consensus.cc:3037] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:07.904464 10640 raft_consensus.cc:2454] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41f43c3af1a4e47812247f3dad6704f in term 1.
I20260925 02:51:07.904482 10782 raft_consensus.cc:2454] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41f43c3af1a4e47812247f3dad6704f in term 1.
I20260925 02:51:07.905004 10853 leader_election.cc:302] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 011f95d37c2c453b937aad8b44cbc700, 448d46b354074cb887780c697d77f80e, e41f43c3af1a4e47812247f3dad6704f; no voters:
I20260925 02:51:07.905360 11018 raft_consensus.cc:2781] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:07.905795 11018 raft_consensus.cc:686] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [term 1 LEADER]: Becoming Leader. State: Replica: e41f43c3af1a4e47812247f3dad6704f, State: Running, Role: LEADER
I20260925 02:51:07.906080 11018 consensus_queue.cc:237] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } }
I20260925 02:51:07.907873 11043 tablet_replica.cc:439] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: TxnStatusTablet state changed. Reason: New leader e41f43c3af1a4e47812247f3dad6704f. Latest consensus state: current_term: 1 leader_uuid: "e41f43c3af1a4e47812247f3dad6704f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } }
I20260925 02:51:07.908169 11043 tablet_replica.cc:442] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:51:07.908802 11045 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:51:07.909051 11045 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:51:07.909479 10482 catalog_manager.cc:5495] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f reported cstate change: term changed from 0 to 1, leader changed from <none> to e41f43c3af1a4e47812247f3dad6704f (127.2.172.67). New cstate: current_term: 1 leader_uuid: "e41f43c3af1a4e47812247f3dad6704f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } health_report { overall_health: UNKNOWN } } }
I20260925 02:51:07.928252 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.921276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:40889 (local address 127.0.0.1:46250)
0925 02:51:07.921670 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:07.921693 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:07.921860 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:07.922401 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:07.922407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:07.922421 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:51:07.922616 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.922626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.924251 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.924267 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.925891 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:07.925905 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.926907 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.926913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:07.927099 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.927171 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:51:07.927675 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:51:07.927702 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:51:07.927788 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:51:07.927940 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260925 02:51:07.928248 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.921220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.2.172.67:40889)
0925 02:51:07.921390 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.921396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.922040 (+ 644us) server_negotiation.cc:407] Connection header received
0925 02:51:07.922115 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.922120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.922241 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.922339 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:07.922796 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.924056 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.926054 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.926740 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.926873 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.927361 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:07.927502 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:07.927657 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:07.927833 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 02:51:07.927916 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:51:07.932746 10782 raft_consensus.cc:1261] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Refusing update from remote peer e41f43c3af1a4e47812247f3dad6704f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:07.932756 10640 raft_consensus.cc:1261] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Refusing update from remote peer e41f43c3af1a4e47812247f3dad6704f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:07.933431 11043 consensus_queue.cc:1035] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [LEADER]: Connected to new peer: Peer: permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:07.933727 11018 consensus_queue.cc:1035] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f [LEADER]: Connected to new peer: Peer: permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:07.953887 11007 tablet_replica.cc:439] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700: TxnStatusTablet state changed. Reason: New leader e41f43c3af1a4e47812247f3dad6704f. Latest consensus state: current_term: 1 leader_uuid: "e41f43c3af1a4e47812247f3dad6704f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } }
I20260925 02:51:07.954219 11007 tablet_replica.cc:442] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:51:07.955508 11005 tablet_replica.cc:439] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e: TxnStatusTablet state changed. Reason: New leader e41f43c3af1a4e47812247f3dad6704f. Latest consensus state: current_term: 1 leader_uuid: "e41f43c3af1a4e47812247f3dad6704f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } }
I20260925 02:51:07.955868 11005 tablet_replica.cc:442] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:51:07.957647 11047 mvcc.cc:205] Tried to move back new op lower bound from 7333087919846084608 to 7333087919745748992. Current Snapshot: MvccSnapshot[applied={T|T < 7333087919846084608}]
I20260925 02:51:07.960672 11048 mvcc.cc:205] Tried to move back new op lower bound from 7333087919846084608 to 7333087919745748992. Current Snapshot: MvccSnapshot[applied={T|T < 7333087919846084608}]
I20260925 02:51:07.964974 11007 tablet_replica.cc:439] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e41f43c3af1a4e47812247f3dad6704f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } }
I20260925 02:51:07.965368 11007 tablet_replica.cc:442] T 807515a54db44eef86511e8564d3d8b7 P 011f95d37c2c453b937aad8b44cbc700: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:51:07.970067 11043 tablet_replica.cc:439] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e41f43c3af1a4e47812247f3dad6704f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } }
I20260925 02:51:07.970392 11043 tablet_replica.cc:442] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:51:07.973311 11043 tablet_replica.cc:439] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e41f43c3af1a4e47812247f3dad6704f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } }
I20260925 02:51:07.973593 11043 tablet_replica.cc:442] T 807515a54db44eef86511e8564d3d8b7 P e41f43c3af1a4e47812247f3dad6704f: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:51:07.978042 11005 tablet_replica.cc:439] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e41f43c3af1a4e47812247f3dad6704f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } } peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } }
I20260925 02:51:07.978313 11005 tablet_replica.cc:442] T 807515a54db44eef86511e8564d3d8b7 P 448d46b354074cb887780c697d77f80e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:51:07.986835 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.980340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.2.172.66:35319)
0925 02:51:07.980991 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.980996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.981043 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:51:07.981133 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.981145 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.981222 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.981409 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:07.981931 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.982761 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.984299 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:07.984817 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:07.984982 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:07.985470 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:07.986132 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:07.986304 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:07.986388 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:51:07.986494 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":330,"threads_started":1}
I20260925 02:51:08.005491 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.998688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:44691 (local address 127.2.172.67:40889)
0925 02:51:07.998966 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:07.998975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:07.999487 (+ 512us) server_negotiation.cc:407] Connection header received
0925 02:51:07.999755 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:07.999760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:07.999843 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:07.999925 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:08.000509 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.001362 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.003433 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.004083 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.004199 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:08.004759 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:08.004874 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:08.004950 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:08.005108 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 02:51:08.005177 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:51:08.005622 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:07.998656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:40889 (local address 127.2.172.66:44691)
0925 02:51:07.999363 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:07.999385 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:07.999572 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:07.999966 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:07.999972 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:07.999986 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:51:08.000350 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:08.000363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.001501 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.001507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:08.003268 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:08.003285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.004282 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.004289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:08.004468 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:08.004552 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:51:08.004978 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:51:08.004990 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 02:51:08.005078 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:51:08.005226 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":369,"threads_started":1}
I20260925 02:51:08.016728 10928 raft_consensus.cc:1261] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Refusing update from remote peer 011f95d37c2c453b937aad8b44cbc700: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:08.016820 10640 raft_consensus.cc:1261] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Refusing update from remote peer 011f95d37c2c453b937aad8b44cbc700: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:08.017588 11031 consensus_queue.cc:1035] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [LEADER]: Connected to new peer: Peer: permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:08.018028 11007 consensus_queue.cc:1035] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:08.030294 11049 mvcc.cc:205] Tried to move back new op lower bound from 7333087920187695104 to 7333087919003615232. Current Snapshot: MvccSnapshot[applied={T|T < 7333087920187695104}]
I20260925 02:51:08.058105 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:08.051183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:39507 (local address 127.2.172.66:35319)
0925 02:51:08.051387 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:08.051395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:08.051854 (+ 459us) server_negotiation.cc:407] Connection header received
0925 02:51:08.052057 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:08.052062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:08.052141 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:08.052239 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:08.052791 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.053933 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.055785 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.056481 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.056624 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:08.057278 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:08.057429 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:08.057526 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:08.057698 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 02:51:08.057783 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:51:08.058372 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:08.050987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:35319 (local address 127.2.172.67:39507)
0925 02:51:08.051699 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:08.051756 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:08.051924 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:08.052254 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:08.052259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:08.052271 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:51:08.052619 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:08.052631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.054085 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.054090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:08.055635 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:08.055651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.056752 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.056759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:08.057004 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:08.057079 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:51:08.057552 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:51:08.057577 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:51:08.057671 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:51:08.057823 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":416,"threads_started":1}
I20260925 02:51:08.108254 10480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:08.108569 10480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:08.112373 10480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f: Sending DeleteTablet for 3 replicas of tablet b94f0306b4d84517972d3621cb7382cf
I20260925 02:51:08.113302 10620 tablet_service.cc:1514] Processing DeleteTablet for tablet b94f0306b4d84517972d3621cb7382cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:08 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:45646
I20260925 02:51:08.113482 10762 tablet_service.cc:1514] Processing DeleteTablet for tablet b94f0306b4d84517972d3621cb7382cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:08 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:60942
I20260925 02:51:08.114432 10903 tablet_service.cc:1514] Processing DeleteTablet for tablet b94f0306b4d84517972d3621cb7382cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:08 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:46234
I20260925 02:51:08.127249 11070 tablet_replica.cc:330] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e: stopping tablet replica
I20260925 02:51:08.127660 11070 raft_consensus.cc:2229] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:08.128046 11070 raft_consensus.cc:2258] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:08.129321 11070 ts_tablet_manager.cc:1905] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:08.136920 11070 ts_tablet_manager.cc:1918] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:51:08.137207 11070 log.cc:1196] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/b94f0306b4d84517972d3621cb7382cf
I20260925 02:51:08.147563 11070 ts_tablet_manager.cc:1939] T b94f0306b4d84517972d3621cb7382cf P 448d46b354074cb887780c697d77f80e: Deleting consensus metadata
I20260925 02:51:08.153847 11072 tablet_replica.cc:330] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f: stopping tablet replica
I20260925 02:51:08.154109 11072 raft_consensus.cc:2229] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:08.154480 11072 raft_consensus.cc:2258] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:08.155514 10466 catalog_manager.cc:4841] TS 448d46b354074cb887780c697d77f80e (127.2.172.65:34873): tablet b94f0306b4d84517972d3621cb7382cf (table test-table [id=e83393dbe64e46e18d9604f1e0ca1bf7]) successfully deleted
I20260925 02:51:08.156348 11072 ts_tablet_manager.cc:1905] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:08.157599 11071 tablet_replica.cc:330] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700: stopping tablet replica
I20260925 02:51:08.157820 11071 raft_consensus.cc:2229] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:51:08.158504 11071 raft_consensus.cc:2258] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:08.165951 11072 ts_tablet_manager.cc:1918] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:51:08.166132 11072 log.cc:1196] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/b94f0306b4d84517972d3621cb7382cf
I20260925 02:51:08.166662 11072 ts_tablet_manager.cc:1939] T b94f0306b4d84517972d3621cb7382cf P e41f43c3af1a4e47812247f3dad6704f: Deleting consensus metadata
I20260925 02:51:08.168093 10467 catalog_manager.cc:4841] TS e41f43c3af1a4e47812247f3dad6704f (127.2.172.67:40889): tablet b94f0306b4d84517972d3621cb7382cf (table test-table [id=e83393dbe64e46e18d9604f1e0ca1bf7]) successfully deleted
I20260925 02:51:08.169378 11071 ts_tablet_manager.cc:1905] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:08.177692 11071 ts_tablet_manager.cc:1918] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:51:08.177863 11071 log.cc:1196] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/b94f0306b4d84517972d3621cb7382cf
I20260925 02:51:08.178388 11071 ts_tablet_manager.cc:1939] T b94f0306b4d84517972d3621cb7382cf P 011f95d37c2c453b937aad8b44cbc700: Deleting consensus metadata
I20260925 02:51:08.179673 10469 catalog_manager.cc:4841] TS 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319): tablet b94f0306b4d84517972d3621cb7382cf (table test-table [id=e83393dbe64e46e18d9604f1e0ca1bf7]) successfully deleted
Sep 25 02:51:08 dist-test-slave-g1xh krb5kdc[10437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:51:08 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:08 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:51:08.220178 11038 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:08.207933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.2.172.126:39009)
0925 02:51:08.208123 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:08.208130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:08.208831 (+ 701us) server_negotiation.cc:407] Connection header received
0925 02:51:08.209077 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:08.209083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:08.209162 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:08.209280 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:08.210628 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.211411 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.212816 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.213424 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.213570 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:08.219914 (+ 6344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43312: BlockingRecv error: recv got EOF from 127.0.0.1:43312 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 02:51:08 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM
I20260925 02:51:08.238794 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:08.223853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.2.172.126:39009)
0925 02:51:08.224030 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:08.224038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:08.224687 (+ 649us) server_negotiation.cc:407] Connection header received
0925 02:51:08.224893 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:08.224898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:08.224975 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:08.225084 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:08.226441 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.227202 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.228564 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.229009 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.229150 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:08.233072 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:08.233078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:08.233081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:08.233118 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:08.235961 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:08.236706 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:08.236711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:08.236713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:08.236855 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:08.237274 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:08.237280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:08.237285 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:08.237565 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:08.237857 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:08.238168 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 02:51:08.238420 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:51:08.242722 10480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:51:08.243417 10480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:51:08.250594 10620 tablet_service.cc:1467] Processing CreateTablet for tablet aa4a0a26d0484c73b40c7cf1e25b9f11 (DEFAULT_TABLE table=test-table [id=fc7bd07c286b41dfbaf4c4625f5d9aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:08.250943 10762 tablet_service.cc:1467] Processing CreateTablet for tablet aa4a0a26d0484c73b40c7cf1e25b9f11 (DEFAULT_TABLE table=test-table [id=fc7bd07c286b41dfbaf4c4625f5d9aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:08.251107 10620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4a0a26d0484c73b40c7cf1e25b9f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:08.251394 10762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4a0a26d0484c73b40c7cf1e25b9f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:08.252208 10903 tablet_service.cc:1467] Processing CreateTablet for tablet aa4a0a26d0484c73b40c7cf1e25b9f11 (DEFAULT_TABLE table=test-table [id=fc7bd07c286b41dfbaf4c4625f5d9aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:08.252648 10903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4a0a26d0484c73b40c7cf1e25b9f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:08.258108 10998 tablet_bootstrap.cc:492] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e: Bootstrap starting.
I20260925 02:51:08.259876 10998 tablet_bootstrap.cc:654] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:08.260195 10999 tablet_bootstrap.cc:492] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f: Bootstrap starting.
I20260925 02:51:08.262212 10999 tablet_bootstrap.cc:654] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:08.262892 11000 tablet_bootstrap.cc:492] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700: Bootstrap starting.
I20260925 02:51:08.265074 11000 tablet_bootstrap.cc:654] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:08.275197 11000 tablet_bootstrap.cc:492] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700: No bootstrap required, opened a new log
I20260925 02:51:08.275408 11000 ts_tablet_manager.cc:1397] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260925 02:51:08.276441 11000 raft_consensus.cc:348] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.276732 11000 raft_consensus.cc:374] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:08.276835 11000 raft_consensus.cc:729] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011f95d37c2c453b937aad8b44cbc700, State: Initialized, Role: FOLLOWER
I20260925 02:51:08.277123 11000 consensus_queue.cc:260] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.280845 11000 ts_tablet_manager.cc:1428] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:51:08.283380 10998 tablet_bootstrap.cc:492] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e: No bootstrap required, opened a new log
I20260925 02:51:08.283607 10998 ts_tablet_manager.cc:1397] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.005s
I20260925 02:51:08.284494 10998 raft_consensus.cc:348] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.284768 10998 raft_consensus.cc:374] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:08.284904 10998 raft_consensus.cc:729] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448d46b354074cb887780c697d77f80e, State: Initialized, Role: FOLLOWER
I20260925 02:51:08.285180 10998 consensus_queue.cc:260] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.286799 10999 tablet_bootstrap.cc:492] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f: No bootstrap required, opened a new log
I20260925 02:51:08.286975 10999 ts_tablet_manager.cc:1397] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.009s
I20260925 02:51:08.287750 10998 ts_tablet_manager.cc:1428] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260925 02:51:08.287912 10999 raft_consensus.cc:348] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.288203 10999 raft_consensus.cc:374] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:08.288306 10999 raft_consensus.cc:729] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41f43c3af1a4e47812247f3dad6704f, State: Initialized, Role: FOLLOWER
I20260925 02:51:08.288517 10999 consensus_queue.cc:260] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.289100 10999 ts_tablet_manager.cc:1428] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:51:08.556200 11005 raft_consensus.cc:482] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:08.556444 11005 raft_consensus.cc:504] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.558040 11005 leader_election.cc:288] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319), e41f43c3af1a4e47812247f3dad6704f (127.2.172.67:40889)
I20260925 02:51:08.579115 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:08.559105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:46649 (local address 127.2.172.67:40889)
0925 02:51:08.572020 (+ 12915us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:08.572027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:08.572939 (+ 912us) server_negotiation.cc:407] Connection header received
0925 02:51:08.573235 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:08.573241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:08.573338 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:08.573456 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:08.574134 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.575027 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.576936 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.578014 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.578197 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:08.578282 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:08.578628 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 02:51:08.578780 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12739,"thread_start_us":12602,"threads_started":1}
I20260925 02:51:08.579744 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:08.558373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:40889 (local address 127.2.172.65:46649)
0925 02:51:08.572795 (+ 14422us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:08.572815 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:08.572998 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:08.573512 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:08.573517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:08.573543 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:08.573925 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:08.573938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.575161 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.575167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:08.576765 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:08.576778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.578214 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.578221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:08.578423 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:08.578467 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 02:51:08.578582 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:51:08.578751 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14302,"mutex_wait_us":6,"thread_start_us":13589,"threads_started":1}
I20260925 02:51:08.580380 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:08.558248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:35319 (local address 127.2.172.65:37781)
0925 02:51:08.572319 (+ 14071us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:08.572351 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:08.572569 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:08.573026 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:08.573031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:08.573058 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:08.573411 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:08.573428 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.574857 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.574863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:08.577178 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:08.577194 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.579302 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.579310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:08.579501 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:08.579530 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 02:51:08.579699 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 02:51:08.579852 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13881,"mutex_wait_us":46,"thread_start_us":14118,"threads_started":1}
I20260925 02:51:08.580399 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:08.558793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:37781 (local address 127.2.172.66:35319)
0925 02:51:08.569300 (+ 10507us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:08.569311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:08.572597 (+ 3286us) server_negotiation.cc:407] Connection header received
0925 02:51:08.572707 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:08.572717 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:08.572818 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:08.572956 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:08.573605 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.574665 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.577367 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.578731 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.579062 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:08.579147 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:08.579726 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 02:51:08.579871 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10332,"thread_start_us":10118,"threads_started":1}
I20260925 02:51:08.580861 10928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4a0a26d0484c73b40c7cf1e25b9f11" candidate_uuid: "448d46b354074cb887780c697d77f80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41f43c3af1a4e47812247f3dad6704f" is_pre_election: true
I20260925 02:51:08.581147 10928 raft_consensus.cc:2454] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 448d46b354074cb887780c697d77f80e in term 0.
I20260925 02:51:08.581180 10782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4a0a26d0484c73b40c7cf1e25b9f11" candidate_uuid: "448d46b354074cb887780c697d77f80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011f95d37c2c453b937aad8b44cbc700" is_pre_election: true
I20260925 02:51:08.581499 10782 raft_consensus.cc:2454] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 448d46b354074cb887780c697d77f80e in term 0.
I20260925 02:51:08.581883 10571 leader_election.cc:302] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448d46b354074cb887780c697d77f80e, e41f43c3af1a4e47812247f3dad6704f; no voters:
I20260925 02:51:08.582413 11005 raft_consensus.cc:2781] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:08.582544 11005 raft_consensus.cc:482] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:08.582665 11005 raft_consensus.cc:3037] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:08.585171 11005 raft_consensus.cc:504] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.585916 11005 leader_election.cc:288] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [CANDIDATE]: Term 1 election: Requested vote from peers 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319), e41f43c3af1a4e47812247f3dad6704f (127.2.172.67:40889)
I20260925 02:51:08.586516 10928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4a0a26d0484c73b40c7cf1e25b9f11" candidate_uuid: "448d46b354074cb887780c697d77f80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41f43c3af1a4e47812247f3dad6704f"
I20260925 02:51:08.586577 10782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4a0a26d0484c73b40c7cf1e25b9f11" candidate_uuid: "448d46b354074cb887780c697d77f80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011f95d37c2c453b937aad8b44cbc700"
I20260925 02:51:08.586787 10928 raft_consensus.cc:3037] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:08.586826 10782 raft_consensus.cc:3037] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:08.589494 10928 raft_consensus.cc:2454] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 448d46b354074cb887780c697d77f80e in term 1.
I20260925 02:51:08.590042 10571 leader_election.cc:302] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448d46b354074cb887780c697d77f80e, e41f43c3af1a4e47812247f3dad6704f; no voters:
I20260925 02:51:08.590109 10782 raft_consensus.cc:2454] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 448d46b354074cb887780c697d77f80e in term 1.
I20260925 02:51:08.590378 11005 raft_consensus.cc:2781] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:08.590847 11005 raft_consensus.cc:686] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 1 LEADER]: Becoming Leader. State: Replica: 448d46b354074cb887780c697d77f80e, State: Running, Role: LEADER
I20260925 02:51:08.591249 11005 consensus_queue.cc:237] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.595749 10482 catalog_manager.cc:5495] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 448d46b354074cb887780c697d77f80e (127.2.172.65). New cstate: current_term: 1 leader_uuid: "448d46b354074cb887780c697d77f80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260925 02:51:08.649333 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:08.642525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.2.172.65:34873)
0925 02:51:08.643129 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:08.643133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:08.643160 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:51:08.643253 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:08.643257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:08.643340 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:08.643484 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:08.644020 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.644926 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.646748 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:08.647202 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:08.647316 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:08.647914 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:08.648530 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:08.648662 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:08.648804 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 02:51:08.648881 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":350,"threads_started":1}
I20260925 02:51:08.653918 10928 raft_consensus.cc:1261] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Refusing update from remote peer 448d46b354074cb887780c697d77f80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:08.654392 10782 raft_consensus.cc:1261] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Refusing update from remote peer 448d46b354074cb887780c697d77f80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:08.654719 11005 consensus_queue.cc:1035] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:08.655051 11092 consensus_queue.cc:1035] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:08.684317 10481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:08.684593 10481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:08.687731 10481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f: Sending DeleteTablet for 3 replicas of tablet aa4a0a26d0484c73b40c7cf1e25b9f11
I20260925 02:51:08.688549 10620 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4a0a26d0484c73b40c7cf1e25b9f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:08 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:45646
I20260925 02:51:08.688700 10762 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4a0a26d0484c73b40c7cf1e25b9f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:08 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:60942
I20260925 02:51:08.688967 10903 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4a0a26d0484c73b40c7cf1e25b9f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:08 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:46234
I20260925 02:51:08.690727 10481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:51:08.691368 10481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:51:08.700416 11106 tablet_replica.cc:330] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e: stopping tablet replica
I20260925 02:51:08.701377 11106 raft_consensus.cc:2229] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:51:08.702107 10619 tablet_service.cc:1467] Processing CreateTablet for tablet 3ecda321fc2a4e1ab98cc7c310d156ca (DEFAULT_TABLE table=test-table [id=fd4d8fa683af46c5bb62e8cc2c0742df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:08.702557 10761 tablet_service.cc:1467] Processing CreateTablet for tablet 3ecda321fc2a4e1ab98cc7c310d156ca (DEFAULT_TABLE table=test-table [id=fd4d8fa683af46c5bb62e8cc2c0742df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:08.702744 10619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ecda321fc2a4e1ab98cc7c310d156ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:08.702980 10761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ecda321fc2a4e1ab98cc7c310d156ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:08.708047 10902 tablet_service.cc:1467] Processing CreateTablet for tablet 3ecda321fc2a4e1ab98cc7c310d156ca (DEFAULT_TABLE table=test-table [id=fd4d8fa683af46c5bb62e8cc2c0742df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:08.708529 10902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ecda321fc2a4e1ab98cc7c310d156ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:08.709643 10998 tablet_bootstrap.cc:492] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e: Bootstrap starting.
I20260925 02:51:08.711437 10998 tablet_bootstrap.cc:654] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:08.724087 11000 tablet_bootstrap.cc:492] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700: Bootstrap starting.
I20260925 02:51:08.724728 10999 tablet_bootstrap.cc:492] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f: Bootstrap starting.
I20260925 02:51:08.725970 11000 tablet_bootstrap.cc:654] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:08.726557 10999 tablet_bootstrap.cc:654] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:08.732120 11106 raft_consensus.cc:2258] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:08.733727 10998 tablet_bootstrap.cc:492] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e: No bootstrap required, opened a new log
I20260925 02:51:08.734016 10998 ts_tablet_manager.cc:1397] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.006s
I20260925 02:51:08.735234 10998 raft_consensus.cc:348] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.735687 10998 raft_consensus.cc:374] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:08.735828 10998 raft_consensus.cc:729] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448d46b354074cb887780c697d77f80e, State: Initialized, Role: FOLLOWER
I20260925 02:51:08.736218 10998 consensus_queue.cc:260] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.736857 10998 ts_tablet_manager.cc:1428] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 02:51:08.739135 11106 ts_tablet_manager.cc:1905] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:08.740797 11107 tablet_replica.cc:330] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700: stopping tablet replica
I20260925 02:51:08.742190 11107 raft_consensus.cc:2229] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:08.742857 11107 raft_consensus.cc:2258] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:08.744376 11107 ts_tablet_manager.cc:1905] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:08.744925 11108 tablet_replica.cc:330] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f: stopping tablet replica
I20260925 02:51:08.745168 11108 raft_consensus.cc:2229] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:08.750381 11107 ts_tablet_manager.cc:1918] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:08.750550 11107 log.cc:1196] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/aa4a0a26d0484c73b40c7cf1e25b9f11
I20260925 02:51:08.750993 11107 ts_tablet_manager.cc:1939] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 011f95d37c2c453b937aad8b44cbc700: Deleting consensus metadata
I20260925 02:51:08.751104 11106 ts_tablet_manager.cc:1918] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:08.751266 11106 log.cc:1196] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/aa4a0a26d0484c73b40c7cf1e25b9f11
I20260925 02:51:08.744673 11000 tablet_bootstrap.cc:492] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700: No bootstrap required, opened a new log
I20260925 02:51:08.745458 10999 tablet_bootstrap.cc:492] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f: No bootstrap required, opened a new log
I20260925 02:51:08.751792 11106 ts_tablet_manager.cc:1939] T aa4a0a26d0484c73b40c7cf1e25b9f11 P 448d46b354074cb887780c697d77f80e: Deleting consensus metadata
I20260925 02:51:08.751994 11000 ts_tablet_manager.cc:1397] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.008s
I20260925 02:51:08.751995 10999 ts_tablet_manager.cc:1397] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.007s
I20260925 02:51:08.752600 11108 raft_consensus.cc:2258] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:08.753065 10466 catalog_manager.cc:4841] TS 448d46b354074cb887780c697d77f80e (127.2.172.65:34873): tablet aa4a0a26d0484c73b40c7cf1e25b9f11 (table test-table [id=fc7bd07c286b41dfbaf4c4625f5d9aaf]) successfully deleted
I20260925 02:51:08.753167 11000 raft_consensus.cc:348] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.753415 11000 raft_consensus.cc:374] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:08.753497 11000 raft_consensus.cc:729] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011f95d37c2c453b937aad8b44cbc700, State: Initialized, Role: FOLLOWER
I20260925 02:51:08.753539 10999 raft_consensus.cc:348] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.753780 10999 raft_consensus.cc:374] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:08.753857 10999 raft_consensus.cc:729] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41f43c3af1a4e47812247f3dad6704f, State: Initialized, Role: FOLLOWER
I20260925 02:51:08.753762 11000 consensus_queue.cc:260] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.754094 10999 consensus_queue.cc:260] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.754298 11000 ts_tablet_manager.cc:1428] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 02:51:08.754607 10999 ts_tablet_manager.cc:1428] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 02:51:08.757407 11108 ts_tablet_manager.cc:1905] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:08.761170 10469 catalog_manager.cc:4841] TS 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319): tablet aa4a0a26d0484c73b40c7cf1e25b9f11 (table test-table [id=fc7bd07c286b41dfbaf4c4625f5d9aaf]) successfully deleted
I20260925 02:51:08.762400 11108 ts_tablet_manager.cc:1918] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:08.762558 11108 log.cc:1196] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/aa4a0a26d0484c73b40c7cf1e25b9f11
I20260925 02:51:08.763003 11108 ts_tablet_manager.cc:1939] T aa4a0a26d0484c73b40c7cf1e25b9f11 P e41f43c3af1a4e47812247f3dad6704f: Deleting consensus metadata
I20260925 02:51:08.764015 10467 catalog_manager.cc:4841] TS e41f43c3af1a4e47812247f3dad6704f (127.2.172.67:40889): tablet aa4a0a26d0484c73b40c7cf1e25b9f11 (table test-table [id=fc7bd07c286b41dfbaf4c4625f5d9aaf]) successfully deleted
I20260925 02:51:08.990311 11092 raft_consensus.cc:482] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:08.990603 11092 raft_consensus.cc:504] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.991465 11092 leader_election.cc:288] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319), e41f43c3af1a4e47812247f3dad6704f (127.2.172.67:40889)
I20260925 02:51:08.992157 10782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecda321fc2a4e1ab98cc7c310d156ca" candidate_uuid: "448d46b354074cb887780c697d77f80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011f95d37c2c453b937aad8b44cbc700" is_pre_election: true
I20260925 02:51:08.992202 10928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecda321fc2a4e1ab98cc7c310d156ca" candidate_uuid: "448d46b354074cb887780c697d77f80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41f43c3af1a4e47812247f3dad6704f" is_pre_election: true
I20260925 02:51:08.992429 10782 raft_consensus.cc:2454] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 448d46b354074cb887780c697d77f80e in term 0.
I20260925 02:51:08.992453 10928 raft_consensus.cc:2454] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 448d46b354074cb887780c697d77f80e in term 0.
I20260925 02:51:08.992966 10573 leader_election.cc:302] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011f95d37c2c453b937aad8b44cbc700, 448d46b354074cb887780c697d77f80e; no voters:
I20260925 02:51:08.993320 11092 raft_consensus.cc:2781] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:08.993448 11092 raft_consensus.cc:482] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:08.993587 11092 raft_consensus.cc:3037] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:08.996029 11092 raft_consensus.cc:504] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:08.996680 11092 leader_election.cc:288] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [CANDIDATE]: Term 1 election: Requested vote from peers 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319), e41f43c3af1a4e47812247f3dad6704f (127.2.172.67:40889)
I20260925 02:51:08.997216 10782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecda321fc2a4e1ab98cc7c310d156ca" candidate_uuid: "448d46b354074cb887780c697d77f80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011f95d37c2c453b937aad8b44cbc700"
I20260925 02:51:08.997234 10928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecda321fc2a4e1ab98cc7c310d156ca" candidate_uuid: "448d46b354074cb887780c697d77f80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41f43c3af1a4e47812247f3dad6704f"
I20260925 02:51:08.997429 10782 raft_consensus.cc:3037] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:08.997462 10928 raft_consensus.cc:3037] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:08.999461 10782 raft_consensus.cc:2454] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 448d46b354074cb887780c697d77f80e in term 1.
I20260925 02:51:08.999979 10573 leader_election.cc:302] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011f95d37c2c453b937aad8b44cbc700, 448d46b354074cb887780c697d77f80e; no voters:
I20260925 02:51:09.000311 11092 raft_consensus.cc:2781] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:09.000511 11092 raft_consensus.cc:686] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 1 LEADER]: Becoming Leader. State: Replica: 448d46b354074cb887780c697d77f80e, State: Running, Role: LEADER
I20260925 02:51:09.000792 11092 consensus_queue.cc:237] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } }
I20260925 02:51:09.000922 10928 raft_consensus.cc:2454] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 448d46b354074cb887780c697d77f80e in term 1.
I20260925 02:51:09.003904 10481 catalog_manager.cc:5495] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 448d46b354074cb887780c697d77f80e (127.2.172.65). New cstate: current_term: 1 leader_uuid: "448d46b354074cb887780c697d77f80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "448d46b354074cb887780c697d77f80e" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 34873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 } health_report { overall_health: UNKNOWN } } }
I20260925 02:51:09.050774 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:09.044622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:40761 (local address 127.2.172.67:40889)
0925 02:51:09.044854 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:09.044861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:09.045537 (+ 676us) server_negotiation.cc:407] Connection header received
0925 02:51:09.045831 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:09.045836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:09.045904 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:09.046006 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:09.046538 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.047390 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.048946 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.049355 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.049522 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:09.049998 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:09.050107 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:09.050214 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:09.050377 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 02:51:09.050455 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:51:09.050912 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:09.044607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:40889 (local address 127.2.172.65:40761)
0925 02:51:09.045413 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:09.045429 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:09.045649 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:09.046026 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:09.046031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:09.046042 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:51:09.046384 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:09.046392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.047529 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.047534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:09.048791 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:09.048804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.049582 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.049589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:09.049737 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:09.049796 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:51:09.050225 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:51:09.050238 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 02:51:09.050357 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 02:51:09.050495 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":444,"threads_started":1}
I20260925 02:51:09.061233 10928 raft_consensus.cc:1261] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Refusing update from remote peer 448d46b354074cb887780c697d77f80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:09.061313 10782 raft_consensus.cc:1261] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Refusing update from remote peer 448d46b354074cb887780c697d77f80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:09.061882 11104 consensus_queue.cc:1035] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41f43c3af1a4e47812247f3dad6704f" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:09.062561 11104 consensus_queue.cc:1035] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "011f95d37c2c453b937aad8b44cbc700" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:09.096917 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:09.089744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:40511 (local address 127.2.172.65:34873)
0925 02:51:09.089907 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:09.089913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:09.090045 (+ 132us) server_negotiation.cc:407] Connection header received
0925 02:51:09.090353 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:09.090358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:09.090448 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:09.090541 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:09.091269 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.092208 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.094446 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.095209 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.095326 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:09.096045 (+ 719us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:09.096187 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:09.096285 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:09.096480 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 02:51:09.096567 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:51:09.097921 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:09.089314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:34873 (local address 127.2.172.67:40511)
0925 02:51:09.089924 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:09.089945 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:09.090183 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:09.090749 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:09.090754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:09.090767 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:51:09.091079 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:09.091092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.092382 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.092388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:09.094252 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:09.094269 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.095577 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.095586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:09.095789 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:09.095852 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:51:09.096313 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:51:09.096326 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 02:51:09.097387 (+ 1061us) client_negotiation.cc:240] Negotiation successful
0925 02:51:09.097539 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":599,"threads_started":1}
I20260925 02:51:09.132666 10482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:09.132894 10482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:09.136245 10482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82e89f6efe2e4236b72b353319c59a5f: Sending DeleteTablet for 3 replicas of tablet 3ecda321fc2a4e1ab98cc7c310d156ca
I20260925 02:51:09.137177 10620 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ecda321fc2a4e1ab98cc7c310d156ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:09 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:45646
I20260925 02:51:09.137219 10762 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ecda321fc2a4e1ab98cc7c310d156ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:09 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:60942
I20260925 02:51:09.137372 10903 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ecda321fc2a4e1ab98cc7c310d156ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:09 UTC) from {username='oryx', principal='oryx/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:46234
I20260925 02:51:09.137509 11106 tablet_replica.cc:330] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e: stopping tablet replica
I20260925 02:51:09.137668 11108 tablet_replica.cc:330] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f: stopping tablet replica
I20260925 02:51:09.137696 11106 raft_consensus.cc:2229] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:51:09.137878 11108 raft_consensus.cc:2229] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:09.138034 11108 raft_consensus.cc:2258] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:09.138029 11106 raft_consensus.cc:2258] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:09.138288 11107 tablet_replica.cc:330] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700: stopping tablet replica
I20260925 02:51:09.138474 11107 raft_consensus.cc:2229] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:09.138664 11107 raft_consensus.cc:2258] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:09.139081 11108 ts_tablet_manager.cc:1905] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:09.140262 11107 ts_tablet_manager.cc:1905] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:09.146139 11108 ts_tablet_manager.cc:1918] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:51:09.146310 11108 log.cc:1196] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/3ecda321fc2a4e1ab98cc7c310d156ca
I20260925 02:51:09.146525 11106 ts_tablet_manager.cc:1905] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:09.146733 11108 ts_tablet_manager.cc:1939] T 3ecda321fc2a4e1ab98cc7c310d156ca P e41f43c3af1a4e47812247f3dad6704f: Deleting consensus metadata
I20260925 02:51:09.148141 10467 catalog_manager.cc:4841] TS e41f43c3af1a4e47812247f3dad6704f (127.2.172.67:40889): tablet 3ecda321fc2a4e1ab98cc7c310d156ca (table test-table [id=fd4d8fa683af46c5bb62e8cc2c0742df]) successfully deleted
I20260925 02:51:09.154073 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 10554
I20260925 02:51:09.154081 11107 ts_tablet_manager.cc:1918] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:51:09.154246 11107 log.cc:1196] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/3ecda321fc2a4e1ab98cc7c310d156ca
I20260925 02:51:09.154698 11107 ts_tablet_manager.cc:1939] T 3ecda321fc2a4e1ab98cc7c310d156ca P 011f95d37c2c453b937aad8b44cbc700: Deleting consensus metadata
I20260925 02:51:09.156039 10469 catalog_manager.cc:4841] TS 011f95d37c2c453b937aad8b44cbc700 (127.2.172.66:35319): tablet 3ecda321fc2a4e1ab98cc7c310d156ca (table test-table [id=fd4d8fa683af46c5bb62e8cc2c0742df]) successfully deleted
I20260925 02:51:09.156746 11106 ts_tablet_manager.cc:1918] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:51:09.157016 11106 log.cc:1196] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/3ecda321fc2a4e1ab98cc7c310d156ca
I20260925 02:51:09.157488 11106 ts_tablet_manager.cc:1939] T 3ecda321fc2a4e1ab98cc7c310d156ca P 448d46b354074cb887780c697d77f80e: Deleting consensus metadata
I20260925 02:51:09.158558 10466 catalog_manager.cc:4841] TS 448d46b354074cb887780c697d77f80e (127.2.172.65:34873): tablet 3ecda321fc2a4e1ab98cc7c310d156ca (table test-table [id=fd4d8fa683af46c5bb62e8cc2c0742df]) successfully deleted
Sep 25 02:51:09 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.172.65@KRBTEST.COM
I20260925 02:51:09.180491 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:09.155345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.2.172.65:34873)
0925 02:51:09.155526 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:09.155532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:09.159385 (+ 3853us) server_negotiation.cc:407] Connection header received
0925 02:51:09.159533 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:09.159539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:09.159680 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:09.159797 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:09.161411 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.162568 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.164452 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.165118 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.165248 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:09.170355 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:09.170373 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:09.170389 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:09.170429 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:09.175189 (+ 4760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:09.176442 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:09.176451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:09.176453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:09.176623 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:09.177285 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:09.177294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:09.177297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:09.178143 (+ 846us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:09.178578 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:09.179205 (+ 627us) server_negotiation.cc:299] Negotiation successful
0925 02:51:09.179820 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:51:09.181842 10685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:09.322516 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 10554
I20260925 02:51:09.347939 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 10694
Sep 25 02:51:09 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.172.66@KRBTEST.COM
I20260925 02:51:09.365329 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:09.348777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.2.172.66:35319)
0925 02:51:09.349284 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:09.349288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:09.349307 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:51:09.349365 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:09.349369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:09.349437 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:09.349530 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:09.350760 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.351461 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.352778 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.353215 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.353327 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:09.357835 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:09.357857 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:09.357885 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:09.357932 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:09.362040 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:09.362846 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:09.362856 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:09.362860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:09.363050 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:09.363482 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:09.363487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:09.363491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:09.364025 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:09.364364 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:09.364667 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 02:51:09.364950 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":288,"threads_started":1}
I20260925 02:51:09.366101 10828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:09.493525 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 10694
I20260925 02:51:09.518028 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 10837
Sep 25 02:51:09 dist-test-slave-g1xh krb5kdc[10437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.172.67@KRBTEST.COM
I20260925 02:51:09.533651 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:09.532557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:34873 (local address 127.2.172.67:51533)
0925 02:51:09.533096 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:09.533272 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":262,"threads_started":1}
W20260925 02:51:09.534036 10853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.172.65:34873: connect: Connection refused (error 111)
I20260925 02:51:09.534243 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:09.519117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.2.172.67:40889)
0925 02:51:09.519298 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:09.519303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:09.519326 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:51:09.519405 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:09.519410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:09.519471 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:09.519596 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:09.520895 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.521625 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.522995 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:09.523452 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:09.523635 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:09.528309 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:09.528326 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:09.528343 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:09.528390 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:09.531104 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:09.531898 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:09.531911 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:09.531914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:09.532090 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:09.532500 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:09.532504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:09.532506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:09.533014 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:09.533341 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:09.533656 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 02:51:09.533921 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:51:09.534852 10968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:09.689493 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 10837
I20260925 02:51:09.717319 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 10450
I20260925 02:51:09.718609 10522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:09.837348 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 10450
2026-09-25T02:51:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7588 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:51:09 dist-test-slave-g1xh krb5kdc[11135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:51:09 dist-test-slave-g1xh krb5kdc[11135](info): set up 2 sockets
Sep 25 02:51:09 dist-test-slave-g1xh krb5kdc[11135](info): commencing operation
krb5kdc: starting...
W20260925 02:51:10.970573 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:51:11 dist-test-slave-g1xh krb5kdc[11135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:51:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:51:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:51:11.215907 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:34621
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:34621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:11.549474 11148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:11.549863 11148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:11.550086 11148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:11.561169 11148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:11.561339 11148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:11.561440 11148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:11.561545 11148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:11.561636 11148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:11.572397 11148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:34621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:34621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:11.573793 11148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:11.575152 11148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:11.588611 11153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:11.588588 11154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:11.588567 11156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:11.589911 11148 server_base.cc:1034] running on GCE node
I20260925 02:51:11.590816 11148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:11.592537 11148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:11.593817 11148 hybrid_clock.cc:648] HybridClock initialized: now 1790304671593743 us; error 54 us; skew 500 ppm
I20260925 02:51:11.594252 11148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T02:51:11Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1836 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:51:11 dist-test-slave-g1xh krb5kdc[11162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:51:11 dist-test-slave-g1xh krb5kdc[11162](info): set up 2 sockets
Sep 25 02:51:11 dist-test-slave-g1xh krb5kdc[11162](info): commencing operation
krb5kdc: starting...
W20260925 02:51:12.802459 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:51:12 dist-test-slave-g1xh krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:51:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:51:13.039384 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:38135
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:38135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:13.390439 11176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:13.390777 11176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:13.390962 11176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:13.399118 11176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:13.399246 11176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:13.399294 11176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:13.399391 11176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:13.399482 11176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:13.409858 11176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:38135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:38135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:13.411350 11176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:13.412745 11176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:13.424714 11184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:13.424796 11182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:13.424757 11181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:13.425985 11176 server_base.cc:1034] running on GCE node
I20260925 02:51:13.426776 11176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:13.428403 11176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:13.429615 11176 hybrid_clock.cc:648] HybridClock initialized: now 1790304673429563 us; error 38 us; skew 500 ppm
I20260925 02:51:13.429982 11176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:13 dist-test-slave-g1xh krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:13.437625 11176 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:51:13.439254 11176 webserver.cc:460] Webserver started at http://127.2.172.126:38321/ using document root <none> and password file <none>
I20260925 02:51:13.440059 11176 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:13.440234 11176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:13.440567 11176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:13.443761 11176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "7a108e45b17b4b97868d1f1ceb738484"
format_stamp: "Formatted at 2026-09-25 02:51:13 on dist-test-slave-g1xh"
server_key: "0ecbf70200ed446499da236eddb2e864"
server_key_iv: "057b6622ed7141d925106c3b8a5ef095"
server_key_version: "encryptionkey@0"
I20260925 02:51:13.444581 11176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "7a108e45b17b4b97868d1f1ceb738484"
format_stamp: "Formatted at 2026-09-25 02:51:13 on dist-test-slave-g1xh"
server_key: "0ecbf70200ed446499da236eddb2e864"
server_key_iv: "057b6622ed7141d925106c3b8a5ef095"
server_key_version: "encryptionkey@0"
I20260925 02:51:13.449997 11176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:51:13.453580 11191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:13.454938 11176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:51:13.455132 11176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "7a108e45b17b4b97868d1f1ceb738484"
format_stamp: "Formatted at 2026-09-25 02:51:13 on dist-test-slave-g1xh"
server_key: "0ecbf70200ed446499da236eddb2e864"
server_key_iv: "057b6622ed7141d925106c3b8a5ef095"
server_key_version: "encryptionkey@0"
I20260925 02:51:13.455349 11176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:13.499903 11176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:13.504766 11176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:13.505165 11176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:13.526058 11176 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:38135
I20260925 02:51:13.526088 11242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:38135 every 8 connection(s)
I20260925 02:51:13.527858 11176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:51:13.532444 11243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:13.534238 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 11176
I20260925 02:51:13.534521 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:51:13.535421 2737 external_mini_cluster.cc:1428] Setting key 24e1dd282ac76e4eb3f00944f798c24e
I20260925 02:51:13.549170 11243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484: Bootstrap starting.
Sep 25 02:51:13 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:51:13.553952 11243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:13.555423 11243 log.cc:824] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:13.558770 11243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484: No bootstrap required, opened a new log
I20260925 02:51:13.562788 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:13.537861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.2.172.126:38135)
0925 02:51:13.538764 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:13.538776 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:13.538818 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:51:13.540580 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:13.540607 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:13.541057 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:13.541425 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:13.543431 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:13.544982 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:13.546948 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:13.548043 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:13.548289 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:13.553803 (+ 5514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:13.553817 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:13.553835 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:13.553882 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:13.557782 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:13.558810 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:13.558838 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:13.558847 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:13.559047 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:13.559540 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:13.559585 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:13.559591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:13.560361 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:13.560662 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:13.561136 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 02:51:13.561586 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":430,"threads_started":1}
I20260925 02:51:13.568361 11243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a108e45b17b4b97868d1f1ceb738484" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38135 } }
I20260925 02:51:13.568786 11243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:13.568897 11243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a108e45b17b4b97868d1f1ceb738484, State: Initialized, Role: FOLLOWER
I20260925 02:51:13.569514 11243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a108e45b17b4b97868d1f1ceb738484" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38135 } }
I20260925 02:51:13.569813 11243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:51:13.569957 11243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:51:13.570129 11243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:13.573077 11243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a108e45b17b4b97868d1f1ceb738484" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38135 } }
I20260925 02:51:13.573653 11243 leader_election.cc:302] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a108e45b17b4b97868d1f1ceb738484; no voters:
I20260925 02:51:13.574401 11243 leader_election.cc:288] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:51:13.574553 11248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:13.575405 11248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 1 LEADER]: Becoming Leader. State: Replica: 7a108e45b17b4b97868d1f1ceb738484, State: Running, Role: LEADER
I20260925 02:51:13.576375 11248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a108e45b17b4b97868d1f1ceb738484" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38135 } }
I20260925 02:51:13.577054 11243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:51:13.579330 11250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a108e45b17b4b97868d1f1ceb738484. Latest consensus state: current_term: 1 leader_uuid: "7a108e45b17b4b97868d1f1ceb738484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a108e45b17b4b97868d1f1ceb738484" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38135 } } }
I20260925 02:51:13.579861 11250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:13.580039 11249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a108e45b17b4b97868d1f1ceb738484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a108e45b17b4b97868d1f1ceb738484" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38135 } } }
I20260925 02:51:13.580427 11249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:13.584694 11255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:13.590541 11255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:13.601361 11255 catalog_manager.cc:1347] Generated new cluster ID: 6b1ba4cd56fb44818eaac4c0f3685f5f
I20260925 02:51:13.601501 11255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:13.632246 11255 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:51:13.633720 11255 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:13.646114 11255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484: Generated new TSK 0
I20260925 02:51:13.647147 11255 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:51:13.706681 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:38135
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:14.046720 11269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:14.047011 11269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:14.047107 11269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:14.054394 11269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:14.054513 11269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:14.054628 11269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:51:14.065465 11269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:38135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:14.066717 11269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:14.068244 11269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:14.081545 11274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:14.081478 11275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:14.081494 11277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:14.082435 11269 server_base.cc:1034] running on GCE node
I20260925 02:51:14.083156 11269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:14.084378 11269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:14.085638 11269 hybrid_clock.cc:648] HybridClock initialized: now 1790304674085588 us; error 47 us; skew 500 ppm
I20260925 02:51:14.086002 11269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:14 dist-test-slave-g1xh krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304674, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:14.089457 11269 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:51:14.090981 11269 webserver.cc:460] Webserver started at http://127.2.172.65:34299/ using document root <none> and password file <none>
I20260925 02:51:14.091778 11269 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:14.091925 11269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:14.092252 11269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:14.095227 11269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "9e99daaeb52a4371ab862a97751d2a07"
format_stamp: "Formatted at 2026-09-25 02:51:14 on dist-test-slave-g1xh"
server_key: "9d2bb41deff8a8ba6b0b80a8e9cc68ed"
server_key_iv: "9893d08fc703f37143eaf22bfd0eb59e"
server_key_version: "encryptionkey@0"
I20260925 02:51:14.096091 11269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "9e99daaeb52a4371ab862a97751d2a07"
format_stamp: "Formatted at 2026-09-25 02:51:14 on dist-test-slave-g1xh"
server_key: "9d2bb41deff8a8ba6b0b80a8e9cc68ed"
server_key_iv: "9893d08fc703f37143eaf22bfd0eb59e"
server_key_version: "encryptionkey@0"
I20260925 02:51:14.101365 11269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:51:14.105093 11284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:14.106490 11269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:51:14.106693 11269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "9e99daaeb52a4371ab862a97751d2a07"
format_stamp: "Formatted at 2026-09-25 02:51:14 on dist-test-slave-g1xh"
server_key: "9d2bb41deff8a8ba6b0b80a8e9cc68ed"
server_key_iv: "9893d08fc703f37143eaf22bfd0eb59e"
server_key_version: "encryptionkey@0"
I20260925 02:51:14.106918 11269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:14.125229 11269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:14.129491 11269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:14.129796 11269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:14.130858 11269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:14.132254 11269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:14.132356 11269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:14.132508 11269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:14.132580 11269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:14.166661 11269 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:42797
I20260925 02:51:14.166747 11397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:42797 every 8 connection(s)
I20260925 02:51:14.168020 11269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:51:14.170601 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 11269
I20260925 02:51:14.170830 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:51:14.171468 2737 external_mini_cluster.cc:1428] Setting key b7019e37c5d282904121aa82c3e642c7
Sep 25 02:51:14 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304674, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:51:14.193416 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:14.170719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:43259 (local address 127.2.172.126:38135)
0925 02:51:14.173254 (+ 2535us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:14.173259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:14.173278 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:51:14.175358 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:14.175370 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:14.175494 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:14.175666 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:14.177675 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:14.179055 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:14.181110 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:14.181935 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:14.182106 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:14.186779 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:14.186785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:14.186789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:14.186839 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:14.189819 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:14.190750 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:14.190764 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:14.190767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:14.190894 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:14.191413 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:14.191424 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:14.191427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:14.191789 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:14.191958 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:14.192777 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 02:51:14.193025 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2431,"thread_start_us":2326,"threads_started":1}
I20260925 02:51:14.194532 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:14.171346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:43259)
0925 02:51:14.172357 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:14.172424 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:14.174899 (+ 2475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:14.175947 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:14.175958 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:14.176648 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:14.177461 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:14.177483 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:14.179320 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:14.179326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:14.180920 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:14.180937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:14.182245 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:14.182251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:14.182639 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:14.183539 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:14.183621 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:14.186478 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:14.190010 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:14.190021 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:14.190043 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:14.190594 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:14.191022 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:14.191028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:14.191032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:14.191273 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:14.191934 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:14.191943 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:14.192342 (+ 399us) client_negotiation.cc:769] Sending connection context
0925 02:51:14.192646 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 02:51:14.193255 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":416,"threads_started":1}
I20260925 02:51:14.196426 11398 heartbeater.cc:347] Connected to a master server at 127.2.172.126:38135
I20260925 02:51:14.196768 11398 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:14.197551 11398 heartbeater.cc:511] Master 127.2.172.126:38135 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
I20260925 02:51:14.200099 11208 ts_manager.cc:194] Registered new tserver with Master: 9e99daaeb52a4371ab862a97751d2a07 (127.2.172.65:42797)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
I20260925 02:51:14.201952 11208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:43259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:51:14.230314 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:38135
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:14.579411 11404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:14.579811 11404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:14.579917 11404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:14.587946 11404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:14.588131 11404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:14.588336 11404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:51:14.598888 11404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:38135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:14.600373 11404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:14.601738 11404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:14.616075 11412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:14.616159 11409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:14.616137 11410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:14.617192 11404 server_base.cc:1034] running on GCE node
I20260925 02:51:14.617884 11404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:14.618906 11404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:14.620234 11404 hybrid_clock.cc:648] HybridClock initialized: now 1790304674620167 us; error 54 us; skew 500 ppm
I20260925 02:51:14.620666 11404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:14 dist-test-slave-g1xh krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304674, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:14.624572 11404 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:51:14.626307 11404 webserver.cc:460] Webserver started at http://127.2.172.66:45255/ using document root <none> and password file <none>
I20260925 02:51:14.627140 11404 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:14.627307 11404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:14.627749 11404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:14.631083 11404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "2e4688efbdd9469a877b5de83a0ddd4e"
format_stamp: "Formatted at 2026-09-25 02:51:14 on dist-test-slave-g1xh"
server_key: "d8bc3956f0870b5c3e92b772523981cd"
server_key_iv: "f5e49a6bf2b44ca63909a971ce99804e"
server_key_version: "encryptionkey@0"
I20260925 02:51:14.632061 11404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "2e4688efbdd9469a877b5de83a0ddd4e"
format_stamp: "Formatted at 2026-09-25 02:51:14 on dist-test-slave-g1xh"
server_key: "d8bc3956f0870b5c3e92b772523981cd"
server_key_iv: "f5e49a6bf2b44ca63909a971ce99804e"
server_key_version: "encryptionkey@0"
I20260925 02:51:14.637996 11404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 02:51:14.641839 11419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:14.643384 11404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:51:14.643637 11404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "2e4688efbdd9469a877b5de83a0ddd4e"
format_stamp: "Formatted at 2026-09-25 02:51:14 on dist-test-slave-g1xh"
server_key: "d8bc3956f0870b5c3e92b772523981cd"
server_key_iv: "f5e49a6bf2b44ca63909a971ce99804e"
server_key_version: "encryptionkey@0"
I20260925 02:51:14.643869 11404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:14.658775 11404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:14.663393 11404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:14.663751 11404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:14.664754 11404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:14.666116 11404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:14.666285 11404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:14.666435 11404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:14.666517 11404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:14.702117 11404 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:34185
I20260925 02:51:14.702175 11532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:34185 every 8 connection(s)
I20260925 02:51:14.703490 11404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:51:14.705441 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 11404
I20260925 02:51:14.705699 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:51:14.706230 2737 external_mini_cluster.cc:1428] Setting key f296137cdaad217614b89d587813abe7
Sep 25 02:51:14 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304674, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:51:14.730309 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:14.706667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:38457 (local address 127.2.172.126:38135)
0925 02:51:14.707182 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:14.707187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:14.708497 (+ 1310us) server_negotiation.cc:407] Connection header received
0925 02:51:14.711807 (+ 3310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:14.711813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:14.711941 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:14.712042 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:14.714411 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:14.715161 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:14.718059 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:14.718597 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:14.718725 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:14.723694 (+ 4969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:14.723701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:14.723704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:14.723746 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:14.726669 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:14.727630 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:14.727643 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:14.727648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:14.727785 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:14.728283 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:14.728293 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:14.728295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:14.728588 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:14.728803 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:14.729638 (+ 835us) server_negotiation.cc:299] Negotiation successful
0925 02:51:14.729890 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":250,"threads_started":1}
I20260925 02:51:14.731407 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:14.707268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:38457)
0925 02:51:14.708291 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:14.708347 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:14.711399 (+ 3052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:14.712361 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:14.712377 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:14.713269 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:14.714202 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:14.714235 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:14.715337 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:14.715348 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:14.717850 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:14.717871 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:14.718891 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:14.718901 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:14.719312 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:14.720376 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:14.720402 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:14.723307 (+ 2905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:14.726906 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:14.726917 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:14.726952 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:14.727453 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:14.727915 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:14.727920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:14.727925 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:14.728134 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:14.728739 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:14.728748 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:14.729172 (+ 424us) client_negotiation.cc:769] Sending connection context
0925 02:51:14.729532 (+ 360us) client_negotiation.cc:240] Negotiation successful
0925 02:51:14.730150 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":378,"threads_started":1}
I20260925 02:51:14.733331 11533 heartbeater.cc:347] Connected to a master server at 127.2.172.126:38135
I20260925 02:51:14.733697 11533 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
I20260925 02:51:14.734717 11533 heartbeater.cc:511] Master 127.2.172.126:38135 requested a full tablet report, sending...
I20260925 02:51:14.736593 11208 ts_manager.cc:194] Registered new tserver with Master: 2e4688efbdd9469a877b5de83a0ddd4e (127.2.172.66:34185)
I20260925 02:51:14.737597 11208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:38457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:51:14.771899 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:38135
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:15.114708 11539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:15.114997 11539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:15.115172 11539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:15.122577 11539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:15.122697 11539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:15.122886 11539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:51:15.133009 11539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:38135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:15.134435 11539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:15.135787 11539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:15.149241 11544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:15.149256 11545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:15.149533 11547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:15.150092 11539 server_base.cc:1034] running on GCE node
I20260925 02:51:15.150738 11539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:15.151808 11539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:15.153045 11539 hybrid_clock.cc:648] HybridClock initialized: now 1790304675153000 us; error 57 us; skew 500 ppm
I20260925 02:51:15.153422 11539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:15 dist-test-slave-g1xh krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:15.156855 11539 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:51:15.158373 11539 webserver.cc:460] Webserver started at http://127.2.172.67:40481/ using document root <none> and password file <none>
I20260925 02:51:15.159161 11539 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:15.159274 11539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:15.159538 11539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:15.162675 11539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "2f3e9a42617849c19469470fac026aa4"
format_stamp: "Formatted at 2026-09-25 02:51:15 on dist-test-slave-g1xh"
server_key: "1b2e63263e2f60d3685983b398d6ba81"
server_key_iv: "6501a5d594689d2285ee227811c9496c"
server_key_version: "encryptionkey@0"
I20260925 02:51:15.163465 11539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "2f3e9a42617849c19469470fac026aa4"
format_stamp: "Formatted at 2026-09-25 02:51:15 on dist-test-slave-g1xh"
server_key: "1b2e63263e2f60d3685983b398d6ba81"
server_key_iv: "6501a5d594689d2285ee227811c9496c"
server_key_version: "encryptionkey@0"
I20260925 02:51:15.168929 11539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 02:51:15.172555 11554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:15.174283 11539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:51:15.174472 11539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "2f3e9a42617849c19469470fac026aa4"
format_stamp: "Formatted at 2026-09-25 02:51:15 on dist-test-slave-g1xh"
server_key: "1b2e63263e2f60d3685983b398d6ba81"
server_key_iv: "6501a5d594689d2285ee227811c9496c"
server_key_version: "encryptionkey@0"
I20260925 02:51:15.174681 11539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:15.199904 11539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:15.204468 11539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:15.204771 11539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:15.205206 11398 heartbeater.cc:503] Master 127.2.172.126:38135 was elected leader, sending a full tablet report...
I20260925 02:51:15.205938 11539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:15.207934 11539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:15.208042 11539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:15.208186 11539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:15.208271 11539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:15.245533 11539 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:39223
I20260925 02:51:15.245630 11667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:39223 every 8 connection(s)
I20260925 02:51:15.246861 11539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:51:15.256722 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 11539
I20260925 02:51:15.256927 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:51:15.257565 2737 external_mini_cluster.cc:1428] Setting key 3104490c14054af94273a999b2fc90ab
Sep 25 02:51:15 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:51:15.267268 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.249662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:46875 (local address 127.2.172.126:38135)
0925 02:51:15.250126 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.250133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.251172 (+ 1039us) server_negotiation.cc:407] Connection header received
0925 02:51:15.253086 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.253091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.253171 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.253275 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:15.254765 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.255651 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.257323 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.257954 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.258125 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.261887 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:15.261892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:15.261896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:15.261936 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:15.264618 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.265336 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.265340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.265342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.265457 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.265834 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.265837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.265839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.266094 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:15.266230 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.266844 (+ 614us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.267001 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260925 02:51:15.268265 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.250207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:46875)
0925 02:51:15.250998 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:15.251040 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:15.252808 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:15.253498 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:15.253507 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:15.254020 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:15.254579 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.254596 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.255792 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.255796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.257181 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.257193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.258224 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.258230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.258448 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.259220 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:15.259259 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:15.261692 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:15.264754 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:15.264761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:15.264774 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:15.265227 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:15.265553 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:15.265556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:15.265558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:15.265741 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:15.266200 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:15.266205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:15.266507 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 02:51:15.266746 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 02:51:15.267122 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":368,"threads_started":1}
I20260925 02:51:15.269614 11668 heartbeater.cc:347] Connected to a master server at 127.2.172.126:38135
I20260925 02:51:15.269944 11668 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:15.270741 11668 heartbeater.cc:511] Master 127.2.172.126:38135 requested a full tablet report, sending...
I20260925 02:51:15.272413 11208 ts_manager.cc:194] Registered new tserver with Master: 2f3e9a42617849c19469470fac026aa4 (127.2.172.67:39223)
I20260925 02:51:15.273427 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:51:15.273451 11208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:46875
WARNING: no policy specified for oryx/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab.
Entry for principal oryx/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab.
WARNING: no policy specified for oryx/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab.
Entry for principal oryx/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab.
WARNING: no policy specified for oryx/127.2.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab.
Entry for principal oryx/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab.
I20260925 02:51:15.428362 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 11269
Sep 25 02:51:15 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:51:15.445533 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.429581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.2.172.65:42797)
0925 02:51:15.430201 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.430208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.430235 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:51:15.430300 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.430307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.430513 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.430641 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:15.432146 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.433334 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.434706 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.435318 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.435425 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.439880 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:15.439892 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:15.439907 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:15.439941 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:15.442749 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.443431 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.443437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.443440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.443628 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.443996 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.444000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.444002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.444332 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:15.444557 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.444863 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.445056 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":297,"threads_started":1}
I20260925 02:51:15.446975 11393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:15.533752 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 11269
I20260925 02:51:15.556622 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 11404
Sep 25 02:51:15 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:51:15.577232 11681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.557731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.2.172.66:34185)
0925 02:51:15.558349 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.558358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.558416 (+ 58us) server_negotiation.cc:407] Connection header received
0925 02:51:15.558510 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.558522 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.558863 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.559172 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:15.561420 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.562984 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.564845 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.565413 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.565571 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.570811 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:15.570825 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:15.570841 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:15.570879 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:15.574057 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.574831 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.574845 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.574847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.574972 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.575380 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.575385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.575390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.575876 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:15.576200 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.576536 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.576829 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":338,"threads_started":1}
I20260925 02:51:15.578864 11528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:15.672698 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 11404
I20260925 02:51:15.692373 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 11539
Sep 25 02:51:15 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:51:15.710155 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.693472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.2.172.67:39223)
0925 02:51:15.693995 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.694003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.694033 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:51:15.694094 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.694101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.694368 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.694550 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:15.696135 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.697214 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.698563 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.699211 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.699384 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.704145 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:15.704157 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:15.704180 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:15.704221 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:15.707091 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.707907 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.707914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.707917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.708045 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.708457 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.708462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.708465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.708820 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:15.709152 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.709484 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.709774 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":253,"threads_started":1}
I20260925 02:51:15.711429 11663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:15.803915 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 11539
I20260925 02:51:15.821406 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 11176
I20260925 02:51:15.822861 11238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:15.928180 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 11176
I20260925 02:51:15.949905 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:38135
--webserver_interface=127.2.172.126
--webserver_port=38321
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:38135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:16.283963 11686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:16.284343 11686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:16.284502 11686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:16.292166 11686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:16.292320 11686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:16.292434 11686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:16.292542 11686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:16.292621 11686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:16.302990 11686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:38135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:38135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=38321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:16.304472 11686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:16.305827 11686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:16.317965 11692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:16.318173 11691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:16.318252 11694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:16.319271 11686 server_base.cc:1034] running on GCE node
I20260925 02:51:16.320209 11686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:16.321851 11686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:16.323117 11686 hybrid_clock.cc:648] HybridClock initialized: now 1790304676323046 us; error 56 us; skew 500 ppm
I20260925 02:51:16.323524 11686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:16 dist-test-slave-g1xh krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304676, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:16.326850 11686 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:51:16.328507 11686 webserver.cc:460] Webserver started at http://127.2.172.126:38321/ using document root <none> and password file <none>
I20260925 02:51:16.329242 11686 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:16.329404 11686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:16.335153 11686 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 02:51:16.338414 11701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:16.339967 11686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:51:16.340181 11686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "7a108e45b17b4b97868d1f1ceb738484"
format_stamp: "Formatted at 2026-09-25 02:51:13 on dist-test-slave-g1xh"
server_key: "0ecbf70200ed446499da236eddb2e864"
server_key_iv: "057b6622ed7141d925106c3b8a5ef095"
server_key_version: "encryptionkey@0"
I20260925 02:51:16.341190 11686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:16.371815 11686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:16.376166 11686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:16.376464 11686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:16.396028 11686 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:38135
I20260925 02:51:16.396056 11752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:38135 every 8 connection(s)
I20260925 02:51:16.397636 11686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:51:16.404475 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 11686
I20260925 02:51:16.404826 11753 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:51:16.405324 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:42797
--local_ip_for_outbound_sockets=127.2.172.65
--tserver_master_addrs=127.2.172.126:38135
--webserver_port=34299
--webserver_interface=127.2.172.65
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.172.65
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:51:16.408232 11753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484: Bootstrap starting.
I20260925 02:51:16.427623 11753 log.cc:824] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:16.436722 11753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:51:16.437250 11753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484: Bootstrap complete.
I20260925 02:51:16.446770 11753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a108e45b17b4b97868d1f1ceb738484" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38135 } }
I20260925 02:51:16.447701 11753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a108e45b17b4b97868d1f1ceb738484, State: Initialized, Role: FOLLOWER
I20260925 02:51:16.448324 11753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a108e45b17b4b97868d1f1ceb738484" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38135 } }
I20260925 02:51:16.448598 11753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:51:16.448748 11753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:51:16.448942 11753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:51:16.452986 11753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a108e45b17b4b97868d1f1ceb738484" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38135 } }
I20260925 02:51:16.453529 11753 leader_election.cc:302] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a108e45b17b4b97868d1f1ceb738484; no voters:
I20260925 02:51:16.454242 11753 leader_election.cc:288] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:51:16.454394 11757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:51:16.455950 11757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [term 2 LEADER]: Becoming Leader. State: Replica: 7a108e45b17b4b97868d1f1ceb738484, State: Running, Role: LEADER
I20260925 02:51:16.456612 11757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a108e45b17b4b97868d1f1ceb738484" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38135 } }
I20260925 02:51:16.456820 11753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:51:16.459277 11759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a108e45b17b4b97868d1f1ceb738484. Latest consensus state: current_term: 2 leader_uuid: "7a108e45b17b4b97868d1f1ceb738484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a108e45b17b4b97868d1f1ceb738484" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38135 } } }
I20260925 02:51:16.459868 11758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7a108e45b17b4b97868d1f1ceb738484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a108e45b17b4b97868d1f1ceb738484" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38135 } } }
I20260925 02:51:16.460630 11758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:16.460783 11759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:16.469509 11763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:16.476914 11763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:16.479059 11763 catalog_manager.cc:1259] Loaded cluster ID: 6b1ba4cd56fb44818eaac4c0f3685f5f
I20260925 02:51:16.479684 11763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:16.487684 11763 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:16.490142 11763 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7a108e45b17b4b97868d1f1ceb738484: Loaded TSK: 0
I20260925 02:51:16.492585 11763 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:51:16.782558 11755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:16.782907 11755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:16.783069 11755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:16.791183 11755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:16.791347 11755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:16.791528 11755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:51:16.801883 11755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.65.keytab
--principal=oryx/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:42797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=34299
--tserver_master_addrs=127.2.172.126:38135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:16.803324 11755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:16.804749 11755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:16.820909 11780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:16.820883 11782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:16.821205 11779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:16.821717 11755 server_base.cc:1034] running on GCE node
I20260925 02:51:16.822341 11755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:16.823330 11755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:16.824566 11755 hybrid_clock.cc:648] HybridClock initialized: now 1790304676824497 us; error 62 us; skew 500 ppm
I20260925 02:51:16.824891 11755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:16 dist-test-slave-g1xh krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304676, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:16.828316 11755 init.cc:377] Logged in from keytab as oryx/127.2.172.65@KRBTEST.COM (short username oryx)
I20260925 02:51:16.830060 11755 webserver.cc:460] Webserver started at http://127.2.172.65:34299/ using document root <none> and password file <none>
I20260925 02:51:16.830813 11755 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:16.830976 11755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:16.836885 11755 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:51:16.839772 11789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:16.841244 11755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:51:16.841446 11755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "9e99daaeb52a4371ab862a97751d2a07"
format_stamp: "Formatted at 2026-09-25 02:51:14 on dist-test-slave-g1xh"
server_key: "9d2bb41deff8a8ba6b0b80a8e9cc68ed"
server_key_iv: "9893d08fc703f37143eaf22bfd0eb59e"
server_key_version: "encryptionkey@0"
I20260925 02:51:16.842419 11755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:16.876693 11755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:16.881265 11755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:16.881554 11755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:16.882560 11755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:16.883987 11755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:16.884073 11755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:16.884243 11755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:16.884341 11755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:16.918208 11755 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:42797
I20260925 02:51:16.918300 11901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:42797 every 8 connection(s)
I20260925 02:51:16.920010 11755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:51:16.920670 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 11755
I20260925 02:51:16.921456 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:34185
--local_ip_for_outbound_sockets=127.2.172.66
--tserver_master_addrs=127.2.172.126:38135
--webserver_port=45255
--webserver_interface=127.2.172.66
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.172.66
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:51:16 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:16 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:51:17.284859 11906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:17.285202 11906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:17.285367 11906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:17.292824 11906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:17.292994 11906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:17.293176 11906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:51:17.302999 11906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.66.keytab
--principal=oryx/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:34185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=45255
--tserver_master_addrs=127.2.172.126:38135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:17.304390 11906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:17.305666 11906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:17.323787 11912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:17.324012 11913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:17.325143 11915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:17.325552 11906 server_base.cc:1034] running on GCE node
I20260925 02:51:17.326226 11906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:17.327303 11906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:17.328682 11906 hybrid_clock.cc:648] HybridClock initialized: now 1790304677328587 us; error 103 us; skew 500 ppm
I20260925 02:51:17.329025 11906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:17 dist-test-slave-g1xh krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304677, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:17.332305 11906 init.cc:377] Logged in from keytab as oryx/127.2.172.66@KRBTEST.COM (short username oryx)
I20260925 02:51:17.333927 11906 webserver.cc:460] Webserver started at http://127.2.172.66:45255/ using document root <none> and password file <none>
I20260925 02:51:17.334618 11906 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:17.334761 11906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:17.341207 11906 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:51:17.344264 11922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:17.346158 11906 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:51:17.346338 11906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "2e4688efbdd9469a877b5de83a0ddd4e"
format_stamp: "Formatted at 2026-09-25 02:51:14 on dist-test-slave-g1xh"
server_key: "d8bc3956f0870b5c3e92b772523981cd"
server_key_iv: "f5e49a6bf2b44ca63909a971ce99804e"
server_key_version: "encryptionkey@0"
I20260925 02:51:17.347366 11906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:17.370185 11906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:17.374686 11906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:17.374985 11906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:17.376117 11906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:17.377616 11906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:17.377749 11906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:17.377885 11906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:17.377974 11906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:17.424189 11906 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:34185
I20260925 02:51:17.424244 12034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:34185 every 8 connection(s)
I20260925 02:51:17.425623 11906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:51:17.429458 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 11906
I20260925 02:51:17.430198 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:39223
--local_ip_for_outbound_sockets=127.2.172.67
--tserver_master_addrs=127.2.172.126:38135
--webserver_port=40481
--webserver_interface=127.2.172.67
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.172.67
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:51:17 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:17 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:51:18.123912 12039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:18.124344 12039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:18.124511 12039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:18.138680 12039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:18.138876 12039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:18.139071 12039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:51:18.155279 12039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/krb5kdc/oryx_127.2.172.67.keytab
--principal=oryx/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:39223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=40481
--tserver_master_addrs=127.2.172.126:38135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:18.157099 12039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:18.158845 12039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:18.182354 12045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:18.188905 12046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:18.189548 12048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:18.189934 12039 server_base.cc:1034] running on GCE node
I20260925 02:51:18.190615 12039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:18.191776 12039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:18.193041 12039 hybrid_clock.cc:648] HybridClock initialized: now 1790304678192965 us; error 66 us; skew 500 ppm
I20260925 02:51:18.193461 12039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:18 dist-test-slave-g1xh krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304678, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:18.196909 12039 init.cc:377] Logged in from keytab as oryx/127.2.172.67@KRBTEST.COM (short username oryx)
I20260925 02:51:18.200551 12039 webserver.cc:460] Webserver started at http://127.2.172.67:40481/ using document root <none> and password file <none>
I20260925 02:51:18.201311 12039 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:18.201478 12039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:18.207924 12039 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 02:51:18.210844 12055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:18.212368 12039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:51:18.212652 12039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "2f3e9a42617849c19469470fac026aa4"
format_stamp: "Formatted at 2026-09-25 02:51:15 on dist-test-slave-g1xh"
server_key: "1b2e63263e2f60d3685983b398d6ba81"
server_key_iv: "6501a5d594689d2285ee227811c9496c"
server_key_version: "encryptionkey@0"
I20260925 02:51:18.213639 12039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:18.234380 12039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:18.239277 12039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:18.239671 12039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:18.240881 12039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:18.242465 12039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:18.242584 12039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:18.242708 12039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:18.242782 12039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260925 02:51:18.243695 11904 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.308s user 0.007s sys 0.009s
I20260925 02:51:18.246182 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:16.923370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:39981)
0925 02:51:16.924240 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:16.924285 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:16.926024 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:16.929268 (+ 3244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:16.929279 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:16.929901 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:16.930455 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:16.930472 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:16.932217 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:16.932225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:16.933426 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:16.933439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:16.934461 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:16.934468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:16.934745 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:16.935504 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:16.935527 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:18.244669 (+1309142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":394,"threads_started":1}
W20260925 02:51:18.246680 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:16.923439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:39981 (local address 127.2.172.126:38135)
0925 02:51:16.928011 (+ 4572us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:16.928024 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:16.928064 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:51:16.928378 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:16.928390 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:16.928720 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:16.929087 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:16.930688 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:16.932027 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:16.933583 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:16.934215 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:16.934413 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:18.245364 (+1310951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:39981: BlockingRecv error: recv got EOF from 127.2.172.65:39981 (error 108)
Metrics: {"server-negotiator.queue_time_us":4358,"thread_start_us":4131,"threads_started":1}
W20260925 02:51:18.247879 11902 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:38135 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.172.126:38135: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Sep 25 02:51:18 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:18 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:18.265831 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:18.248769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:50697)
0925 02:51:18.249205 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:18.249229 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:18.249355 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:18.250297 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:18.250306 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:18.251011 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:18.251320 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:18.251338 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.253605 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.253615 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:18.256094 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:18.256123 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.257974 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.257984 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:18.258268 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:18.259246 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:18.259279 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:18.265365 (+ 6086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272}
W20260925 02:51:18.266175 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:18.249341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:50697 (local address 127.2.172.126:38135)
0925 02:51:18.249578 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:18.249588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:18.249619 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:51:18.249710 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:18.249719 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:18.249845 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:18.249999 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:18.251618 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.253306 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.256416 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.257411 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.257587 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:18.265754 (+ 8167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:50697: BlockingRecv error: recv got EOF from 127.2.172.65:50697 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 02:51:18 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:18 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:18.281680 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:18.266862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:58721)
0925 02:51:18.267298 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:18.267324 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:18.267459 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:18.268270 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:18.268278 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:18.268811 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:18.269074 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:18.269087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.270861 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.270869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:18.272875 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:18.272898 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.274339 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.274345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:18.274621 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:18.275438 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:18.275467 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:18.281262 (+ 5795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279}
W20260925 02:51:18.282637 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:18.267261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:58721 (local address 127.2.172.126:38135)
0925 02:51:18.267499 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:18.267508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:18.267538 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:51:18.267831 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:18.267841 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:18.267963 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:18.268067 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:18.269282 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.270644 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.273098 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.273882 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.273995 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:18.282294 (+ 8299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:58721: BlockingRecv error: recv got EOF from 127.2.172.65:58721 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260925 02:51:18.283191 11902 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:51:18.297335 12039 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:39223
I20260925 02:51:18.297518 12167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:39223 every 8 connection(s)
I20260925 02:51:18.298874 12039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:51:18.308853 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 12039
Sep 25 02:51:18 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:18 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:18.341295 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:18.309825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.2.172.126:38135)
0925 02:51:18.326124 (+ 16299us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:18.326129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:18.326146 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:51:18.326196 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:18.326200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:18.326262 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:18.326403 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:18.328230 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.329126 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.330515 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.330970 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.331125 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:18.334355 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:18.334365 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:18.334385 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:18.334425 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:18.337635 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:18.338378 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:18.338386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:18.338393 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:18.338568 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:18.338994 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:18.338999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:18.339001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:18.339672 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:18.340000 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:18.340448 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 02:51:18.340989 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16176,"thread_start_us":16032,"threads_started":1}
W20260925 02:51:18.612520 12038 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.170s user 0.004s sys 0.011s
W20260925 02:51:18.614189 12037 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:17.428552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:34981 (local address 127.2.172.126:38135)
0925 02:51:17.429218 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:17.429228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:17.430414 (+ 1186us) server_negotiation.cc:407] Connection header received
0925 02:51:17.434502 (+ 4088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:17.434511 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:17.434621 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:17.434746 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:17.437117 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:17.438990 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:17.440786 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:17.441188 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:17.441330 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:18.613943 (+1172613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:34981: BlockingRecv error: recv got EOF from 127.2.172.66:34981 (error 108)
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":279,"threads_started":1}
I20260925 02:51:18.615468 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:17.429053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:34981)
0925 02:51:17.430168 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:17.430251 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:17.434050 (+ 3799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:17.435217 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:17.435231 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:17.436041 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:17.436848 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:17.436877 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:17.439210 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:17.439214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:17.440608 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:17.440621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:17.441425 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:17.441435 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:17.441678 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:17.442416 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:17.442439 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:18.613823 (+1171384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":411,"threads_started":1}
W20260925 02:51:18.616456 12035 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:38135 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.172.126:38135: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Sep 25 02:51:18 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:18 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:18.634862 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:18.617128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:40895)
0925 02:51:18.617399 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:18.617428 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:18.617620 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:18.618027 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:18.618034 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:18.618754 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:18.619057 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:18.619074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.620352 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.620363 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:18.621698 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:18.621714 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.622447 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.622453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:18.622621 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:18.630358 (+ 7737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:18.630378 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:18.634605 (+ 4227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 02:51:18.635319 12037 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:18.617237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:40895 (local address 127.2.172.126:38135)
0925 02:51:18.617391 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:18.617396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:18.617518 (+ 122us) server_negotiation.cc:407] Connection header received
0925 02:51:18.617784 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:18.617790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:18.617866 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:18.617946 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:18.619222 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.620189 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.621841 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.622262 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.622398 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:18.634968 (+ 12570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:40895: BlockingRecv error: recv got EOF from 127.2.172.66:40895 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 02:51:18 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:18 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:51:18.648136 12037 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:18.636005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:46821 (local address 127.2.172.126:38135)
0925 02:51:18.636174 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:18.636178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:18.636192 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:51:18.636432 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:18.636441 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:18.636550 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:18.636702 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:18.637304 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.639023 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.640493 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.641507 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.641706 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:18.647653 (+ 5947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:46821: BlockingRecv error: recv got EOF from 127.2.172.66:46821 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:51:18.648940 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:18.635578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:46821)
0925 02:51:18.635825 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:18.635845 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:18.636272 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:18.636717 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:18.636721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:18.637008 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:18.637157 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:18.637165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.639193 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.639197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:18.640339 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:18.640352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.641703 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.641708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:18.641909 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:18.642604 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:18.642621 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:18.648514 (+ 5893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114}
W20260925 02:51:18.649307 12035 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 02:51:19 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:19 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:19.298990 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:19.284022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:33773)
0925 02:51:19.287807 (+ 3785us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:19.287824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:19.287955 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:19.288478 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:19.288481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:19.288982 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:19.289226 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:19.289236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.291311 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.291315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:19.292371 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:19.292383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.293704 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.293709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:19.293829 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:19.294574 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:19.294590 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:19.298652 (+ 4062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3657,"thread_start_us":3525,"threads_started":1}
W20260925 02:51:19.299670 12175 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:19.284132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:33773 (local address 127.2.172.126:38135)
0925 02:51:19.287638 (+ 3506us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:19.287649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:19.287925 (+ 276us) server_negotiation.cc:407] Connection header received
0925 02:51:19.288132 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:19.288142 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:19.288274 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:19.288440 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:19.289431 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.291127 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.292534 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.293513 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.293690 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:19.299041 (+ 5351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:33773: BlockingRecv error: recv got EOF from 127.2.172.65:33773 (error 108)
Metrics: {"server-negotiator.queue_time_us":3270,"thread_start_us":3106,"threads_started":1}
W20260925 02:51:19.442827 12170 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.122s user 0.007s sys 0.020s
W20260925 02:51:19.443913 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:18.302458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:43413 (local address 127.2.172.126:38135)
0925 02:51:18.302701 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:18.302707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:18.304250 (+ 1543us) server_negotiation.cc:407] Connection header received
0925 02:51:18.308262 (+ 4012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:18.308267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:18.308370 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:18.308493 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:18.311499 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.313175 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.316895 (+ 3720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.317998 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.318136 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:19.443694 (+1125558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:43413: BlockingRecv error: recv got EOF from 127.2.172.67:43413 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:51:19.444502 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:18.302925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:43413)
0925 02:51:18.304013 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:18.304087 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:18.307815 (+ 3728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:18.308909 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:18.308927 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:18.309997 (+ 1070us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:18.311172 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:18.311207 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.313522 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.313531 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:18.316609 (+ 3078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:18.316630 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:18.318646 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:18.318657 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:18.319163 (+ 506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:18.320311 (+ 1148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:18.320367 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:19.443499 (+1123132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":359,"threads_started":1}
W20260925 02:51:19.445360 12168 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:38135 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.172.126:38135: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Sep 25 02:51:19 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:19 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:19.454722 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:19.446011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:34939)
0925 02:51:19.446240 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:19.446256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:19.446390 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:19.446698 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:19.446701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:19.447029 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:19.447194 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:19.447203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.448309 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.448313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:19.449273 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:19.449285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.449953 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.449957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:19.450072 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:19.450686 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:19.450703 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:19.454495 (+ 3792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260925 02:51:19.455042 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:19.446085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:34939 (local address 127.2.172.126:38135)
0925 02:51:19.446292 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:19.446296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:19.446313 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:51:19.446506 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:19.446509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:19.446567 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:19.446643 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:19.447361 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.448177 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.449405 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.449792 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.449881 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:19.454742 (+ 4861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:34939: BlockingRecv error: recv got EOF from 127.2.172.67:34939 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 25 02:51:19 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:19 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:19.464557 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:19.455399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:39615)
0925 02:51:19.455579 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:19.455598 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:19.455715 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:19.455987 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:19.455990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:19.456275 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:19.456423 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:19.456430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.457769 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.457773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:19.458798 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:19.458810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.459632 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.459637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:19.459777 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:19.460418 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:19.460436 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:19.464265 (+ 3829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:51:19.464730 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:19.455441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:39615 (local address 127.2.172.126:38135)
0925 02:51:19.455663 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:19.455667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:19.455683 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:51:19.455826 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:19.455830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:19.455885 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:19.455963 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:19.456611 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.457635 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.458944 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.459442 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.459615 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:19.464461 (+ 4846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:39615: BlockingRecv error: recv got EOF from 127.2.172.67:39615 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 02:51:19.464991 12168 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 02:51:19 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:19 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:19.662461 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:19.650267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:58935)
0925 02:51:19.650986 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:19.651012 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:19.651174 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:19.651691 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:19.651695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:19.652362 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:19.652605 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:19.652626 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.654105 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.654111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:19.655224 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:19.655238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.656125 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.656131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:19.656272 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:19.657108 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:19.657129 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:19.662055 (+ 4926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":373,"threads_started":1}
W20260925 02:51:19.662693 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:19.650653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:58935 (local address 127.2.172.126:38135)
0925 02:51:19.650921 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:19.650930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:19.651103 (+ 173us) server_negotiation.cc:407] Connection header received
0925 02:51:19.651357 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:19.651362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:19.651439 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:19.651652 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:19.652772 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.653936 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.655367 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:19.655916 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:19.656023 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:19.662375 (+ 6352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:58935: BlockingRecv error: recv got EOF from 127.2.172.66:58935 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 02:51:20 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:20 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:20.315749 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:20.304656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:45159)
0925 02:51:20.305305 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:20.305324 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:20.305466 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:20.305863 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:20.305882 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:20.306343 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:20.306574 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:20.306584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:20.307830 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:20.307835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:20.308852 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:20.308893 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:20.309914 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:20.309921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:20.310122 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:20.310978 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:20.311002 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:20.315351 (+ 4349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":380,"threads_started":1}
W20260925 02:51:20.316031 12178 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:20.304719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:45159 (local address 127.2.172.126:38135)
0925 02:51:20.305317 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:20.305322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:20.305416 (+ 94us) server_negotiation.cc:407] Connection header received
0925 02:51:20.305624 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:20.305629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:20.305704 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:20.305809 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:20.306754 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:20.307698 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:20.309068 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:20.309742 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:20.309887 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:20.315712 (+ 5825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:45159: BlockingRecv error: recv got EOF from 127.2.172.65:45159 (error 108)
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":320,"threads_started":1}
Sep 25 02:51:20 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:20 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:20.476899 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:20.465801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:39123)
0925 02:51:20.466575 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:20.466592 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:20.466742 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:20.467121 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:20.467125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:20.467757 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:20.467942 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:20.467951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:20.469130 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:20.469135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:20.470163 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:20.470175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:20.471049 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:20.471055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:20.471197 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:20.472106 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:20.472132 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:20.476491 (+ 4359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":453,"threads_started":1}
W20260925 02:51:20.477031 12178 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:20.465861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:39123 (local address 127.2.172.126:38135)
0925 02:51:20.466094 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:20.466100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:20.466713 (+ 613us) server_negotiation.cc:407] Connection header received
0925 02:51:20.466861 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:20.466865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:20.466958 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:20.467054 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:20.468106 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:20.468982 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:20.470317 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:20.470841 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:20.470955 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:20.476758 (+ 5803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:39123: BlockingRecv error: recv got EOF from 127.2.172.67:39123 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 25 02:51:20 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:20 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:20.675446 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:20.663867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:58417)
0925 02:51:20.664453 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:20.664471 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:20.664629 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:20.665003 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:20.665007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:20.665539 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:20.665756 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:20.665767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:20.667087 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:20.667095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:20.668274 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:20.668289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:20.669228 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:20.669234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:20.669453 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:20.670277 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:20.670298 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:20.675025 (+ 4727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":320,"threads_started":1}
W20260925 02:51:20.675745 12178 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:20.664028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:58417 (local address 127.2.172.126:38135)
0925 02:51:20.664337 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:20.664345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:20.664568 (+ 223us) server_negotiation.cc:407] Connection header received
0925 02:51:20.664749 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:20.664753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:20.664829 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:20.664936 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:20.665964 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:20.666902 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:20.668454 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:20.669058 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:20.669208 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:20.675301 (+ 6093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:58417: BlockingRecv error: recv got EOF from 127.2.172.66:58417 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 25 02:51:21 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:21 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:21.327361 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:21.316795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:51329)
0925 02:51:21.317443 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:21.317471 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:21.317618 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:21.318015 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:21.318018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:21.318446 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:21.318643 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:21.318652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:21.319879 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:21.319883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:21.320867 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:21.320880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:21.321758 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:21.321762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:21.321935 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:21.322754 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:21.322775 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:21.327029 (+ 4254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":304,"threads_started":1}
W20260925 02:51:21.328138 12182 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:21.317019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:51329 (local address 127.2.172.126:38135)
0925 02:51:21.317676 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:21.317685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:21.317709 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:51:21.317779 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:21.317784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:21.317868 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:21.317967 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:21.318861 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:21.319728 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:21.321020 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:21.321599 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:21.321706 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:21.327826 (+ 6120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:51329: BlockingRecv error: recv got EOF from 127.2.172.65:51329 (error 108)
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":298,"threads_started":1}
Sep 25 02:51:21 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:21 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:21.491961 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:21.478523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:47501)
0925 02:51:21.479297 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:21.479322 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:21.479494 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:21.479931 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:21.479936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:21.480576 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:21.481064 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:21.481084 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:21.482486 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:21.482491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:21.483795 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:21.483818 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:21.484892 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:21.484900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:21.485105 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:21.486111 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:21.486147 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:21.491483 (+ 5336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":451,"threads_started":1}
W20260925 02:51:21.492161 12182 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:21.478765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:47501 (local address 127.2.172.126:38135)
0925 02:51:21.478979 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:21.478984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:21.479460 (+ 476us) server_negotiation.cc:407] Connection header received
0925 02:51:21.479667 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:21.479672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:21.479752 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:21.479853 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:21.481362 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:21.482255 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:21.484093 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:21.484686 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:21.484806 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:21.491880 (+ 7074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:47501: BlockingRecv error: recv got EOF from 127.2.172.67:47501 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 02:51:21 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:21 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:21.689383 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:21.677101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:54757)
0925 02:51:21.677853 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:21.677872 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:21.678039 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:21.678542 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:21.678546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:21.679146 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:21.679617 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:21.679635 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:21.681054 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:21.681061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:21.682601 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:21.682619 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:21.683760 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:21.683765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:21.683935 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:21.684610 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:21.684629 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:21.689085 (+ 4456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":428,"threads_started":1}
W20260925 02:51:21.689749 12182 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:21.677257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:54757 (local address 127.2.172.126:38135)
0925 02:51:21.677505 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:21.677511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:21.677987 (+ 476us) server_negotiation.cc:407] Connection header received
0925 02:51:21.678224 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:21.678229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:21.678326 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:21.678430 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:21.679849 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:21.680896 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:21.682821 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:21.683617 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:21.683734 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:21.689435 (+ 5701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:54757: BlockingRecv error: recv got EOF from 127.2.172.66:54757 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 02:51:22 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:22 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:22.339901 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:22.329041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:55055)
0925 02:51:22.329803 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:22.329831 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:22.330032 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:22.330566 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:22.330572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:22.331128 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:22.331412 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:22.331428 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:22.332615 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:22.332619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:22.333685 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:22.333699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:22.334393 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:22.334399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:22.334544 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:22.335302 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:22.335322 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:22.339527 (+ 4205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":358,"threads_started":1}
W20260925 02:51:22.340225 12186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:22.329161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:55055 (local address 127.2.172.126:38135)
0925 02:51:22.330014 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:22.330018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:22.330036 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:51:22.330199 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:22.330204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:22.330300 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:22.330419 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:22.331617 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:22.332484 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:22.333823 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:22.334258 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:22.334355 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:22.339909 (+ 5554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:55055: BlockingRecv error: recv got EOF from 127.2.172.65:55055 (error 108)
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":562,"threads_started":1}
Sep 25 02:51:22 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:22 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:22.504814 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:22.493316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:54157)
0925 02:51:22.493978 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:22.493997 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:22.494153 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:22.494545 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:22.494550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:22.495003 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:22.495220 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:22.495231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:22.496514 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:22.496518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:22.497558 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:22.497573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:22.498411 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:22.498418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:22.498688 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:22.499591 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:22.499613 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:22.504361 (+ 4748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":390,"threads_started":1}
W20260925 02:51:22.504972 12186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:22.493575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:54157 (local address 127.2.172.126:38135)
0925 02:51:22.493937 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:22.493944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:22.494150 (+ 206us) server_negotiation.cc:407] Connection header received
0925 02:51:22.494267 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:22.494271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:22.494349 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:22.494456 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:22.495415 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:22.496328 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:22.497743 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:22.498203 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:22.498382 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:22.504686 (+ 6304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:54157: BlockingRecv error: recv got EOF from 127.2.172.67:54157 (error 108)
Metrics: {"server-negotiator.queue_time_us":191}
Sep 25 02:51:22 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:22 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:22.700927 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:22.690605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:34767)
0925 02:51:22.691192 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:22.691209 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:22.691338 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:22.691689 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:22.691692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:22.692151 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:22.692347 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:22.692356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:22.693421 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:22.693425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:22.694597 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:22.694610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:22.695799 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:22.695805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:22.695979 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:22.696675 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:22.696694 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:22.700624 (+ 3930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":298,"threads_started":1}
W20260925 02:51:22.701407 12186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:22.690811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:34767 (local address 127.2.172.126:38135)
0925 02:51:22.691060 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:22.691067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:22.691290 (+ 223us) server_negotiation.cc:407] Connection header received
0925 02:51:22.691454 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:22.691458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:22.691565 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:22.691655 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:22.692484 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:22.693287 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:22.694980 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:22.695645 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:22.695788 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:22.700911 (+ 5123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:34767: BlockingRecv error: recv got EOF from 127.2.172.66:34767 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 02:51:23 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:23 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:23.352880 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:23.341338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:37999)
0925 02:51:23.342031 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:23.342049 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:23.342201 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:23.342581 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:23.342585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:23.343108 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:23.343306 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:23.343317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:23.344675 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:23.344680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:23.346125 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:23.346165 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:23.347026 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:23.347031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:23.347353 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:23.348113 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:23.348135 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:23.352583 (+ 4448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":399,"threads_started":1}
W20260925 02:51:23.353111 12190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:23.341386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:37999 (local address 127.2.172.126:38135)
0925 02:51:23.342002 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:23.342006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:23.342159 (+ 153us) server_negotiation.cc:407] Connection header received
0925 02:51:23.342329 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:23.342333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:23.342413 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:23.342515 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:23.343493 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:23.344477 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:23.346339 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:23.346845 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:23.346961 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:23.352849 (+ 5888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:37999: BlockingRecv error: recv got EOF from 127.2.172.65:37999 (error 108)
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":370,"threads_started":1}
Sep 25 02:51:23 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:23 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:23.516505 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:23.506265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:39161)
0925 02:51:23.506929 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:23.506950 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:23.507091 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:23.507434 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:23.507437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:23.507956 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:23.508195 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:23.508207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:23.509424 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:23.509431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:23.510720 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:23.510741 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:23.511623 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:23.511632 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:23.511771 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:23.512453 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:23.512473 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:23.516232 (+ 3759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":414,"threads_started":1}
W20260925 02:51:23.516753 12190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:23.506372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:39161 (local address 127.2.172.126:38135)
0925 02:51:23.506645 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:23.506655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:23.507047 (+ 392us) server_negotiation.cc:407] Connection header received
0925 02:51:23.507214 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:23.507218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:23.507283 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:23.507376 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:23.508364 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:23.509245 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:23.510884 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:23.511365 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:23.511475 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:23.516487 (+ 5012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:39161: BlockingRecv error: recv got EOF from 127.2.172.67:39161 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 02:51:23 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:23 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:23.717824 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:23.702123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:46273)
0925 02:51:23.702797 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:23.702815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:23.703006 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:23.703427 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:23.703435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:23.704030 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:23.704216 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:23.704226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:23.705330 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:23.705336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:23.706317 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:23.706338 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:23.707098 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:23.707104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:23.707315 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:23.712881 (+ 5566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:23.712907 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:23.717394 (+ 4487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":380,"threads_started":1}
W20260925 02:51:23.718060 12190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:23.702327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:46273 (local address 127.2.172.126:38135)
0925 02:51:23.702582 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:23.702592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:23.702912 (+ 320us) server_negotiation.cc:407] Connection header received
0925 02:51:23.703157 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:23.703162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:23.703252 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:23.703360 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:23.704383 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:23.705168 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:23.706473 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:23.706941 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:23.707082 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:23.717761 (+ 10679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:46273: BlockingRecv error: recv got EOF from 127.2.172.66:46273 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 02:51:24 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:24 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:24.365440 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:24.354021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:45917)
0925 02:51:24.354814 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:24.354838 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:24.355009 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:24.355435 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:24.355440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:24.356040 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:24.356297 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:24.356308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:24.357562 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:24.357567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:24.358756 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:24.358773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:24.359604 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:24.359610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:24.359784 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:24.360641 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:24.360661 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:24.365039 (+ 4378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":504,"threads_started":1}
W20260925 02:51:24.365813 12193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:24.354056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:45917 (local address 127.2.172.126:38135)
0925 02:51:24.354709 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:24.354715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:24.354940 (+ 225us) server_negotiation.cc:407] Connection header received
0925 02:51:24.355117 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:24.355121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:24.355206 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:24.355308 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:24.356487 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:24.357397 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:24.358923 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:24.359357 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:24.359473 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:24.365424 (+ 5951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:45917: BlockingRecv error: recv got EOF from 127.2.172.65:45917 (error 108)
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":399,"threads_started":1}
Sep 25 02:51:24 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:24 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:24.529134 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:24.517605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:45303)
0925 02:51:24.518402 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:24.518421 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:24.518566 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:24.519018 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:24.519023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:24.519746 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:24.519945 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:24.519955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:24.521172 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:24.521177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:24.522256 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:24.522272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:24.523084 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:24.523090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:24.523277 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:24.524247 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:24.524270 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:24.528837 (+ 4567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":455,"threads_started":1}
W20260925 02:51:24.529459 12193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:24.517755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:45303 (local address 127.2.172.126:38135)
0925 02:51:24.518029 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:24.518034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:24.518537 (+ 503us) server_negotiation.cc:407] Connection header received
0925 02:51:24.518730 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:24.518735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:24.518814 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:24.518909 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:24.520119 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:24.521002 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:24.522462 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:24.522905 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:24.523012 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:24.529158 (+ 6146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:45303: BlockingRecv error: recv got EOF from 127.2.172.67:45303 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 02:51:24 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:24 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:24.729943 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:24.719177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:37119)
0925 02:51:24.719938 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:24.719957 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:24.720210 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:24.720667 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:24.720675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:24.721491 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:24.721767 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:24.721784 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:24.723056 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:24.723062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:24.724037 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:24.724058 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:24.724765 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:24.724770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:24.724954 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:24.725564 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:24.725585 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:24.729664 (+ 4079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":404,"threads_started":1}
W20260925 02:51:24.730288 12193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:24.719320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:37119 (local address 127.2.172.126:38135)
0925 02:51:24.719686 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:24.719693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:24.720099 (+ 406us) server_negotiation.cc:407] Connection header received
0925 02:51:24.720373 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:24.720377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:24.720466 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:24.720586 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:24.722034 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:24.722884 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:24.724205 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:24.724601 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:24.724745 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:24.729907 (+ 5162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:37119: BlockingRecv error: recv got EOF from 127.2.172.66:37119 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 25 02:51:25 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:25 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:25.377233 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:25.366852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:51883)
0925 02:51:25.367786 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:25.367816 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:25.367943 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:25.368339 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:25.368343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:25.368826 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:25.369011 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:25.369020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:25.370176 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:25.370180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:25.371148 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:25.371160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:25.371975 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:25.371984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:25.372118 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:25.372722 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:25.372743 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:25.376954 (+ 4211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":578,"threads_started":1}
W20260925 02:51:25.377439 12197 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:25.366983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:51883 (local address 127.2.172.126:38135)
0925 02:51:25.367499 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:25.367503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:25.367901 (+ 398us) server_negotiation.cc:407] Connection header received
0925 02:51:25.368051 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:25.368055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:25.368139 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:25.368256 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:25.369189 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:25.370032 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:25.371290 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:25.371801 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:25.371916 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:25.377117 (+ 5201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:51883: BlockingRecv error: recv got EOF from 127.2.172.65:51883 (error 108)
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":311,"threads_started":1}
Sep 25 02:51:25 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:25 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:25.540488 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:25.530255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:59869)
0925 02:51:25.530832 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:25.530848 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:25.531023 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:25.531396 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:25.531401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:25.532065 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:25.532251 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:25.532261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:25.533361 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:25.533367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:25.534312 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:25.534323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:25.535119 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:25.535126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:25.535297 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:25.535966 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:25.535993 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:25.540099 (+ 4106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":316,"threads_started":1}
W20260925 02:51:25.540501 12197 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:25.530391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:59869 (local address 127.2.172.126:38135)
0925 02:51:25.530681 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:25.530687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:25.530937 (+ 250us) server_negotiation.cc:407] Connection header received
0925 02:51:25.531179 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:25.531183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:25.531244 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:25.531350 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:25.532409 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:25.533207 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:25.534466 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:25.534877 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:25.535010 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:25.540296 (+ 5286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:59869: BlockingRecv error: recv got EOF from 127.2.172.67:59869 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 25 02:51:25 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:25 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:25.740633 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:25.730988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:47677)
0925 02:51:25.731681 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:25.731698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:25.731821 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:25.732107 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:25.732111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:25.732518 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:25.732717 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:25.732726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:25.733800 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:25.733803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:25.734775 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:25.734788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:25.735509 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:25.735513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:25.735712 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:25.736419 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:25.736437 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:25.740290 (+ 3853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":309,"threads_started":1}
W20260925 02:51:25.741425 12197 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:25.731253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:47677 (local address 127.2.172.126:38135)
0925 02:51:25.731494 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:25.731500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:25.731779 (+ 279us) server_negotiation.cc:407] Connection header received
0925 02:51:25.731908 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:25.731912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:25.731984 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:25.732064 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:25.732886 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:25.733666 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:25.734906 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:25.735358 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:25.735458 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:25.740880 (+ 5422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:47677: BlockingRecv error: recv got EOF from 127.2.172.66:47677 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 25 02:51:26 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:26 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:26.389727 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:26.378365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:51051)
0925 02:51:26.379063 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:26.379083 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:26.379226 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:26.379654 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:26.379659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:26.380155 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:26.380425 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:26.380438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:26.381781 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:26.381785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:26.382829 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:26.382850 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:26.383773 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:26.383781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:26.383983 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:26.384772 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:26.384792 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:26.389395 (+ 4603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":363,"threads_started":1}
W20260925 02:51:26.389971 12202 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:26.378546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:51051 (local address 127.2.172.126:38135)
0925 02:51:26.379103 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:26.379108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:26.379146 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:51:26.379345 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:26.379350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:26.379420 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:26.379524 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:26.380635 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:26.381612 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:26.382991 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:26.383469 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:26.383647 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:26.389672 (+ 6025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:51051: BlockingRecv error: recv got EOF from 127.2.172.65:51051 (error 108)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":325,"threads_started":1}
Sep 25 02:51:26 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:26 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:26.554638 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:26.541746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:40377)
0925 02:51:26.542786 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:26.542808 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:26.542968 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:26.543443 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:26.543447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:26.544026 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:26.544275 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:26.544285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:26.545950 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:26.545959 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:26.547134 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:26.547151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:26.548301 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:26.548308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:26.548477 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:26.549250 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:26.549273 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:26.554240 (+ 4967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":482,"threads_started":1}
W20260925 02:51:26.554832 12202 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:26.541970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:40377 (local address 127.2.172.126:38135)
0925 02:51:26.542192 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:26.542197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:26.542925 (+ 728us) server_negotiation.cc:407] Connection header received
0925 02:51:26.543131 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:26.543136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:26.543219 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:26.543322 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:26.544505 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:26.545611 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:26.547352 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:26.548089 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:26.548206 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:26.554575 (+ 6369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:40377: BlockingRecv error: recv got EOF from 127.2.172.67:40377 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 02:51:26 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:26 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:26.753436 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:26.742143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:44943)
0925 02:51:26.742797 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:26.742815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:26.743017 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:26.743395 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:26.743400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:26.743941 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:26.744140 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:26.744150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:26.745578 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:26.745584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:26.746736 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:26.746750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:26.747835 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:26.747843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:26.748065 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:26.748933 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:26.748954 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:26.753124 (+ 4170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":363,"threads_started":1}
W20260925 02:51:26.753844 12202 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:26.742200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:44943 (local address 127.2.172.126:38135)
0925 02:51:26.742471 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:26.742478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:26.742926 (+ 448us) server_negotiation.cc:407] Connection header received
0925 02:51:26.743170 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:26.743175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:26.743245 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:26.743357 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:26.744325 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:26.745376 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:26.746887 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:26.747598 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:26.747751 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:26.753460 (+ 5709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:44943: BlockingRecv error: recv got EOF from 127.2.172.66:44943 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 02:51:27 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:27 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:27.405045 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:27.391014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:36387)
0925 02:51:27.391630 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:27.391649 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:27.391769 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:27.392270 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:27.392274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:27.392738 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:27.392907 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:27.392916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:27.394436 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:27.394440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:27.396069 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:27.396104 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:27.397709 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:27.397717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:27.397943 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:27.399009 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:27.399041 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:27.404703 (+ 5662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":271,"threads_started":1}
W20260925 02:51:27.405392 12206 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:27.391130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:36387 (local address 127.2.172.126:38135)
0925 02:51:27.391959 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:27.391963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:27.391980 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:51:27.392034 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:27.392038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:27.392100 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:27.392210 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:27.393112 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:27.394254 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:27.396367 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:27.397475 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:27.397664 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:27.405018 (+ 7354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:36387: BlockingRecv error: recv got EOF from 127.2.172.65:36387 (error 108)
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":585,"threads_started":1}
Sep 25 02:51:27 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:27 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:27.565909 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:27.555995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:32861)
0925 02:51:27.556641 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:27.556663 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:27.556839 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:27.557256 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:27.557261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:27.557736 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:27.557926 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:27.557936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:27.558946 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:27.558950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:27.560019 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:27.560043 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:27.561006 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:27.561016 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:27.561183 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:27.561936 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:27.561961 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:27.565644 (+ 3683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":318,"threads_started":1}
W20260925 02:51:27.566082 12206 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:27.556076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:32861 (local address 127.2.172.126:38135)
0925 02:51:27.556313 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:27.556323 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:27.556754 (+ 431us) server_negotiation.cc:407] Connection header received
0925 02:51:27.556997 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:27.557004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:27.557083 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:27.557195 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:27.558062 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:27.558812 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:27.560222 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:27.560781 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:27.560899 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:27.565868 (+ 4969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:32861: BlockingRecv error: recv got EOF from 127.2.172.67:32861 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 02:51:27 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:27 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:27.764328 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:27.754515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:60441)
0925 02:51:27.755026 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:27.755043 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:27.755163 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:27.755468 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:27.755471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:27.755992 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:27.756153 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:27.756163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:27.757247 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:27.757251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:27.758351 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:27.758363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:27.759283 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:27.759292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:27.759485 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:27.760176 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:27.760215 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:27.763984 (+ 3769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":269,"threads_started":1}
W20260925 02:51:27.764410 12206 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:27.754574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:60441 (local address 127.2.172.126:38135)
0925 02:51:27.754811 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:27.754819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:27.755190 (+ 371us) server_negotiation.cc:407] Connection header received
0925 02:51:27.755270 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:27.755274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:27.755337 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:27.755424 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:27.756304 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:27.757096 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:27.758489 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:27.759098 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:27.759218 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:27.764161 (+ 4943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:60441: BlockingRecv error: recv got EOF from 127.2.172.66:60441 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 02:51:28 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:28 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:28.419135 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:28.406133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:43073)
0925 02:51:28.407165 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:28.407186 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:28.407332 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:28.407876 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:28.407881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:28.408358 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:28.408581 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:28.408592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:28.410492 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:28.410497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:28.411641 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:28.411654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:28.412910 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:28.412914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:28.413042 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:28.413865 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:28.413889 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:28.418610 (+ 4721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":737,"threads_started":1}
W20260925 02:51:28.419446 12210 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:28.406219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:43073 (local address 127.2.172.126:38135)
0925 02:51:28.406943 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:28.406952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:28.407277 (+ 325us) server_negotiation.cc:407] Connection header received
0925 02:51:28.407505 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:28.407514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:28.407650 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:28.407791 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:28.408742 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:28.410311 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:28.411797 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:28.412730 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:28.412892 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:28.418952 (+ 6060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:43073: BlockingRecv error: recv got EOF from 127.2.172.65:43073 (error 108)
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":330,"threads_started":1}
Sep 25 02:51:28 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:28 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:28.579828 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:28.567552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:42239)
0925 02:51:28.568294 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:28.568317 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:28.568480 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:28.569050 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:28.569055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:28.569627 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:28.569833 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:28.569845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:28.571780 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:28.571785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:28.573023 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:28.573050 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:28.574065 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:28.574072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:28.574265 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:28.575193 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:28.575218 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:28.579408 (+ 4190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":379,"threads_started":1}
W20260925 02:51:28.580279 12210 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:28.567653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:42239 (local address 127.2.172.126:38135)
0925 02:51:28.567939 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:28.567950 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:28.568434 (+ 484us) server_negotiation.cc:407] Connection header received
0925 02:51:28.568644 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:28.568653 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:28.568768 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:28.568939 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:28.570038 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:28.571581 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:28.573203 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:28.573875 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:28.574018 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:28.579705 (+ 5687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:42239: BlockingRecv error: recv got EOF from 127.2.172.67:42239 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 02:51:28 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:28 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:28.781548 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:28.765622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:41075)
0925 02:51:28.766380 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:28.766400 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:28.766550 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:28.766962 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:28.766965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:28.767527 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:28.767761 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:28.767772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:28.769163 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:28.769171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:28.770537 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:28.770551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:28.771344 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:28.771350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:28.771483 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:28.776684 (+ 5201us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:28.776707 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:28.781117 (+ 4410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":439,"threads_started":1}
W20260925 02:51:28.781733 12210 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:28.765698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:41075 (local address 127.2.172.126:38135)
0925 02:51:28.765928 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:28.765933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:28.766496 (+ 563us) server_negotiation.cc:407] Connection header received
0925 02:51:28.766701 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:28.766705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:28.766799 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:28.766921 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:28.767937 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:28.768966 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:28.770672 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:28.771139 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:28.771244 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:28.781444 (+ 10200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:41075: BlockingRecv error: recv got EOF from 127.2.172.66:41075 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 02:51:29 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:29 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:29.430171 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:29.420399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:44125)
0925 02:51:29.421052 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:29.421079 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:29.421265 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:29.421786 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:29.421793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:29.422304 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:29.422550 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:29.422560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:29.423695 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:29.423702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:29.424599 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:29.424610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:29.425400 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:29.425406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:29.425599 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:29.426270 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:29.426289 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:29.429924 (+ 3635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":388,"threads_started":1}
W20260925 02:51:29.430549 12214 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:29.420591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:44125 (local address 127.2.172.126:38135)
0925 02:51:29.421425 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:29.421430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:29.421445 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:51:29.421513 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:29.421518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:29.421623 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:29.421747 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:29.422705 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:29.423504 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:29.424740 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:29.425237 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:29.425383 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:29.430182 (+ 4799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:44125: BlockingRecv error: recv got EOF from 127.2.172.65:44125 (error 108)
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":538,"threads_started":1}
Sep 25 02:51:29 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:29 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:29.591728 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:29.581146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:43299)
0925 02:51:29.581725 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:29.581744 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:29.581884 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:29.582315 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:29.582319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:29.582835 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:29.583085 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:29.583096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:29.584420 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:29.584428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:29.585528 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:29.585542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:29.586443 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:29.586448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:29.586568 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:29.587386 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:29.587410 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:29.591381 (+ 3971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":326,"threads_started":1}
W20260925 02:51:29.591820 12214 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:29.581238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:43299 (local address 127.2.172.126:38135)
0925 02:51:29.581517 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:29.581524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:29.581887 (+ 363us) server_negotiation.cc:407] Connection header received
0925 02:51:29.581994 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:29.581998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:29.582065 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:29.582223 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:29.583252 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:29.584276 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:29.585731 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:29.586271 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:29.586391 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:29.591640 (+ 5249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:43299: BlockingRecv error: recv got EOF from 127.2.172.67:43299 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 02:51:29 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:29 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:29.792289 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:29.782731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:40471)
0925 02:51:29.783358 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:29.783383 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:29.783541 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:29.783951 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:29.783955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:29.784422 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:29.784613 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:29.784622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:29.785640 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:29.785648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:29.786582 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:29.786592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:29.787409 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:29.787415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:29.787639 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:29.788269 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:29.788286 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:29.791977 (+ 3691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":377,"threads_started":1}
W20260925 02:51:29.792464 12214 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:29.782901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:40471 (local address 127.2.172.126:38135)
0925 02:51:29.783108 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:29.783116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:29.783462 (+ 346us) server_negotiation.cc:407] Connection header received
0925 02:51:29.783693 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:29.783698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:29.783798 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:29.783920 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:29.784762 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:29.785513 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:29.786730 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:29.787250 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:29.787397 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:29.792185 (+ 4788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:40471: BlockingRecv error: recv got EOF from 127.2.172.66:40471 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 02:51:30 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:30 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:30.440244 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:30.431178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:56917)
0925 02:51:30.431783 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:30.431805 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:30.431932 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:30.432370 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:30.432373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:30.432849 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:30.433018 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:30.433026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:30.434026 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:30.434030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:30.434952 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:30.434963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:30.435755 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:30.435760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:30.435967 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:30.436585 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:30.436607 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:30.439951 (+ 3344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":275,"threads_started":1}
W20260925 02:51:30.440377 12218 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:30.431343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:56917 (local address 127.2.172.126:38135)
0925 02:51:30.432049 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:30.432053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:30.432069 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:51:30.432166 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:30.432169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:30.432224 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:30.432316 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:30.433187 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:30.433910 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:30.435147 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:30.435609 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:30.435743 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:30.440165 (+ 4422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:56917: BlockingRecv error: recv got EOF from 127.2.172.65:56917 (error 108)
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":497,"threads_started":1}
Sep 25 02:51:30 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:30 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:30.602458 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:30.592929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:45021)
0925 02:51:30.593490 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:30.593506 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:30.593636 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:30.593961 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:30.593965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:30.594439 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:30.594622 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:30.594631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:30.595690 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:30.595695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:30.596783 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:30.596796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:30.597584 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:30.597590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:30.597770 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:30.598487 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:30.598506 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:30.602141 (+ 3635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":359,"threads_started":1}
W20260925 02:51:30.602504 12218 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:30.593096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:45021 (local address 127.2.172.126:38135)
0925 02:51:30.593370 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:30.593377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:30.593627 (+ 250us) server_negotiation.cc:407] Connection header received
0925 02:51:30.593733 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:30.593738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:30.593802 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:30.593889 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:30.594762 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:30.595472 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:30.596932 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:30.597440 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:30.597533 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:30.602291 (+ 4758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:45021: BlockingRecv error: recv got EOF from 127.2.172.67:45021 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 25 02:51:30 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:30 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:30.803617 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:30.793449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:43463)
0925 02:51:30.794015 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:30.794034 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:30.794148 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:30.794555 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:30.794558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:30.794996 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:30.795209 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:30.795219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:30.796530 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:30.796534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:30.797941 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:30.797960 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:30.798714 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:30.798719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:30.798857 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:30.799537 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:30.799556 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:30.803227 (+ 3671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":313,"threads_started":1}
W20260925 02:51:30.803745 12218 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:30.793529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:43463 (local address 127.2.172.126:38135)
0925 02:51:30.793717 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:30.793722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:30.794131 (+ 409us) server_negotiation.cc:407] Connection header received
0925 02:51:30.794267 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:30.794274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:30.794369 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:30.794461 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:30.795370 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:30.796403 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:30.798083 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:30.798526 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:30.798636 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:30.803429 (+ 4793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:43463: BlockingRecv error: recv got EOF from 127.2.172.66:43463 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 02:51:31 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:31 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:31.450238 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:31.441231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:37709)
0925 02:51:31.441740 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:31.441758 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:31.441862 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:31.442195 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:31.442199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:31.442589 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:31.442733 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:31.442741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:31.443777 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:31.443781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:31.444685 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:31.444696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:31.445404 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:31.445408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:31.445529 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:31.446248 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:31.446266 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:31.450004 (+ 3738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":280,"threads_started":1}
W20260925 02:51:31.450487 12222 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:31.441339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:37709 (local address 127.2.172.126:38135)
0925 02:51:31.441848 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:31.441853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:31.441870 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:51:31.441969 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:31.441973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:31.442030 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:31.442122 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:31.442859 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:31.443659 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:31.444825 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:31.445272 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:31.445367 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:31.450238 (+ 4871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:37709: BlockingRecv error: recv got EOF from 127.2.172.65:37709 (error 108)
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":314,"threads_started":1}
Sep 25 02:51:31 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:31 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:31.613176 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:31.603566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:48321)
0925 02:51:31.604212 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:31.604228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:31.604359 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:31.604694 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:31.604698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:31.605104 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:31.605330 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:31.605340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:31.606404 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:31.606408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:31.607318 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:31.607329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:31.608179 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:31.608185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:31.608415 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:31.609160 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:31.609186 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:31.612874 (+ 3688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":328,"threads_started":1}
W20260925 02:51:31.613389 12222 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:31.603713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:48321 (local address 127.2.172.126:38135)
0925 02:51:31.603997 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:31.604003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:31.604305 (+ 302us) server_negotiation.cc:407] Connection header received
0925 02:51:31.604470 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:31.604474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:31.604534 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:31.604622 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:31.605465 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:31.606272 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:31.607468 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:31.607939 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:31.608063 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:31.613102 (+ 5039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:48321: BlockingRecv error: recv got EOF from 127.2.172.67:48321 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 02:51:31 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:31 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:31.814177 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:31.804766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:41473)
0925 02:51:31.805287 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:31.805308 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:31.805480 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:31.805910 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:31.805914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:31.806463 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:31.806630 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:31.806640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:31.807942 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:31.807947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:31.808925 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:31.808937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:31.809614 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:31.809618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:31.809737 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:31.810335 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:31.810354 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:31.813913 (+ 3559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":256,"threads_started":1}
W20260925 02:51:31.814368 12222 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:31.805067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:41473 (local address 127.2.172.126:38135)
0925 02:51:31.805289 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:31.805297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:31.805402 (+ 105us) server_negotiation.cc:407] Connection header received
0925 02:51:31.805610 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:31.805620 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:31.805701 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:31.805823 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:31.806772 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:31.807796 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:31.809059 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:31.809483 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:31.809579 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:31.814128 (+ 4549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:41473: BlockingRecv error: recv got EOF from 127.2.172.66:41473 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 02:51:32 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:32 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.65@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:32.461273 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:32.451595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.65:40315)
0925 02:51:32.452198 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:32.452217 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:32.452362 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:32.452742 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:32.452746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:32.453138 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:32.453333 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:32.453342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:32.454407 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:32.454412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:32.455448 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:32.455460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:32.456283 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:32.456288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:32.456424 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:32.457104 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:32.457125 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:32.461012 (+ 3887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":302,"threads_started":1}
W20260925 02:51:32.461621 12226 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:32.451714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:40315 (local address 127.2.172.126:38135)
0925 02:51:32.452222 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:32.452227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:32.452274 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:51:32.452521 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:32.452526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:32.452591 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:32.452691 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:32.453495 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:32.454265 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:32.455631 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:32.456126 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:32.456233 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:32.461297 (+ 5064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.65:40315: BlockingRecv error: recv got EOF from 127.2.172.65:40315 (error 108)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":261,"threads_started":1}
Sep 25 02:51:32 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:32 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.67@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:32.623622 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:32.614219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.67:36919)
0925 02:51:32.614722 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:32.614739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:32.614894 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:32.615227 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:32.615232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:32.615715 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:32.615869 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:32.615877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:32.616904 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:32.616909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:32.617890 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:32.617902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:32.618566 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:32.618572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:32.618772 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:32.619350 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:32.619367 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:32.623253 (+ 3886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":270,"threads_started":1}
W20260925 02:51:32.623761 12226 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:32.614313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:36919 (local address 127.2.172.126:38135)
0925 02:51:32.614628 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:32.614634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:32.614817 (+ 183us) server_negotiation.cc:407] Connection header received
0925 02:51:32.615017 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:32.615023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:32.615091 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:32.615192 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:32.616007 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:32.616753 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:32.618031 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:32.618418 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:32.618547 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:32.623459 (+ 4912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.67:36919: BlockingRecv error: recv got EOF from 127.2.172.67:36919 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Sep 25 02:51:32 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:51:32 dist-test-slave-g1xh krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.172.66@KRBTEST.COM for oryx/127.2.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:51:32.824766 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:32.815450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38135 (local address 127.2.172.66:49033)
0925 02:51:32.815978 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:32.815994 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:32.816117 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:32.816410 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:32.816413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:32.816778 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:32.816974 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:32.816982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:32.818058 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:32.818064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:32.819128 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:32.819146 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:32.820150 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:32.820157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:32.820313 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:32.820882 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:32.820900 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:32.824518 (+ 3618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.172.126:38135: Server oryx/127.2.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":292,"threads_started":1}
W20260925 02:51:32.824947 12226 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:32.815516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:49033 (local address 127.2.172.126:38135)
0925 02:51:32.815764 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:32.815769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:32.816063 (+ 294us) server_negotiation.cc:407] Connection header received
0925 02:51:32.816232 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:32.816236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:32.816298 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:32.816374 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:32.817095 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:32.817901 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:32.819292 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:32.819939 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:32.820078 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:32.824712 (+ 4634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.172.66:49033: BlockingRecv error: recv got EOF from 127.2.172.66:49033 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
I20260925 02:51:33.310019 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 11755
I20260925 02:51:33.323935 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:33.311254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.2.172.65:42797)
0925 02:51:33.311908 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:33.311923 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:33.311951 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:51:33.312020 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:33.312026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:33.312272 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:33.312454 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:33.313986 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:33.315148 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:33.316474 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:33.317366 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:33.317487 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:33.320408 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:33.320418 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:33.320434 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:33.320484 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:33.323286 (+ 2802us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.172.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 02:51:33.323701 (+ 415us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37934: Request ticket server kudu/127.2.172.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":369,"threads_started":1}
W20260925 02:51:33.324127 12230 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37934: Request ticket server kudu/127.2.172.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 02:51:33.324103 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 11755: Not authorized: Client connection negotiation failed: client connection to 127.2.172.65:42797: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.172.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 02:51:33.324332 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 11755
I20260925 02:51:33.346617 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 11906
I20260925 02:51:33.363220 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:33.347992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.2.172.66:34185)
0925 02:51:33.348683 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:33.348693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:33.348725 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:51:33.348812 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:33.348821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:33.349120 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:33.349356 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:33.350758 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:33.352514 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:33.354755 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:33.355613 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:33.355779 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:33.359213 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:33.359225 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:33.359242 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:33.359299 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:33.362602 (+ 3303us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.172.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 02:51:33.362974 (+ 372us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50658: Request ticket server kudu/127.2.172.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":353,"threads_started":1}
W20260925 02:51:33.363319 12231 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50658: Request ticket server kudu/127.2.172.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 02:51:33.363569 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 11906: Not authorized: Client connection negotiation failed: client connection to 127.2.172.66:34185: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.172.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 02:51:33.363816 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 11906
I20260925 02:51:33.383208 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 12039
I20260925 02:51:33.397900 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:33.385039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.2.172.67:39223)
0925 02:51:33.385717 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:33.385725 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:33.385752 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:51:33.385825 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:33.385831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:33.386094 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:33.386387 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:33.388001 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:33.389124 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:33.390704 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:33.391282 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:33.391437 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:33.394154 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:33.394165 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:33.394181 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:33.394227 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:33.397084 (+ 2857us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.172.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 02:51:33.397492 (+ 408us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55890: Request ticket server kudu/127.2.172.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":344,"threads_started":1}
W20260925 02:51:33.397927 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 12039: Not authorized: Client connection negotiation failed: client connection to 127.2.172.67:39223: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.172.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 02:51:33.398025 12232 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55890: Request ticket server kudu/127.2.172.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 02:51:33.398080 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 12039
I20260925 02:51:33.416657 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 11686
I20260925 02:51:33.418035 11748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:33.537469 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 11686
2026-09-25T02:51:33Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21866 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T02:51:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:51:33Z Disabled control of system clock
I20260925 02:51:33.623488 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:34627
--webserver_interface=127.2.172.126
--webserver_port=0
--builtin_ntp_servers=127.2.172.84:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:34627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:51:33.957060 12237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:33.957357 12237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:33.957434 12237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:33.964556 12237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:33.964663 12237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:33.964701 12237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:33.964759 12237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:33.964804 12237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:33.974174 12237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:34627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:34627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:33.975445 12237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:33.977377 12237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:33.989060 12242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:33.989144 12245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:33.989732 12243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:33.990497 12237 server_base.cc:1034] running on GCE node
I20260925 02:51:33.991290 12237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:33.992877 12237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:33.994191 12237 hybrid_clock.cc:648] HybridClock initialized: now 1790304693994087 us; error 87 us; skew 500 ppm
I20260925 02:51:33.994566 12237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:51:34.003216 12237 webserver.cc:460] Webserver started at http://127.2.172.126:45279/ using document root <none> and password file <none>
I20260925 02:51:34.004050 12237 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:34.004199 12237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:34.004580 12237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:34.007679 12237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "2f938dd9103547b9b65c162b6c013d28"
format_stamp: "Formatted at 2026-09-25 02:51:33 on dist-test-slave-g1xh"
server_key: "b960452197d527942875df8a1c65d406"
server_key_iv: "790adeaaef25a28f874cfe01d4741ac3"
server_key_version: "encryptionkey@0"
I20260925 02:51:34.008460 12237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "2f938dd9103547b9b65c162b6c013d28"
format_stamp: "Formatted at 2026-09-25 02:51:33 on dist-test-slave-g1xh"
server_key: "b960452197d527942875df8a1c65d406"
server_key_iv: "790adeaaef25a28f874cfe01d4741ac3"
server_key_version: "encryptionkey@0"
I20260925 02:51:34.013739 12237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:51:34.017262 12251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:34.018568 12237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:51:34.018759 12237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "2f938dd9103547b9b65c162b6c013d28"
format_stamp: "Formatted at 2026-09-25 02:51:33 on dist-test-slave-g1xh"
server_key: "b960452197d527942875df8a1c65d406"
server_key_iv: "790adeaaef25a28f874cfe01d4741ac3"
server_key_version: "encryptionkey@0"
I20260925 02:51:34.018985 12237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:34.039300 12237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:34.040288 12237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:34.040591 12237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:34.059649 12237 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:34627
I20260925 02:51:34.059726 12302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:34627 every 8 connection(s)
I20260925 02:51:34.061050 12237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:51:34.064870 12303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:34.068219 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 12237
I20260925 02:51:34.068444 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:51:34.069259 2737 external_mini_cluster.cc:1428] Setting key 934a6f0bbdff0dbe025ff5a0364ffe2c
I20260925 02:51:34.077523 12303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28: Bootstrap starting.
I20260925 02:51:34.080937 12303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:34.082218 12303 log.cc:824] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:34.083184 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:34.071421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.2.172.126:34627)
0925 02:51:34.072405 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:34.072415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:34.072456 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:51:34.074046 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:34.074075 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:34.074554 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:34.074945 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:34.075612 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:34.076986 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:34.078379 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:34.079072 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:34.079287 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:34.080517 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:34.080525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:34.080540 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:34.081127 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:34.081239 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:34.081264 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:34.081402 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:34.081635 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 02:51:34.081974 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":327,"threads_started":1}
I20260925 02:51:34.085525 12303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28: No bootstrap required, opened a new log
I20260925 02:51:34.093271 12303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f938dd9103547b9b65c162b6c013d28" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34627 } }
I20260925 02:51:34.093675 12303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:34.093786 12303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f938dd9103547b9b65c162b6c013d28, State: Initialized, Role: FOLLOWER
I20260925 02:51:34.094421 12303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f938dd9103547b9b65c162b6c013d28" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34627 } }
I20260925 02:51:34.094718 12303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:51:34.094864 12303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:51:34.095052 12303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:34.097275 12303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f938dd9103547b9b65c162b6c013d28" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34627 } }
I20260925 02:51:34.097762 12303 leader_election.cc:302] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f938dd9103547b9b65c162b6c013d28; no voters:
I20260925 02:51:34.098359 12303 leader_election.cc:288] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:51:34.098527 12308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:34.099324 12308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [term 1 LEADER]: Becoming Leader. State: Replica: 2f938dd9103547b9b65c162b6c013d28, State: Running, Role: LEADER
I20260925 02:51:34.099920 12308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f938dd9103547b9b65c162b6c013d28" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34627 } }
I20260925 02:51:34.100487 12303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:51:34.102651 12309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f938dd9103547b9b65c162b6c013d28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f938dd9103547b9b65c162b6c013d28" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34627 } } }
I20260925 02:51:34.102957 12309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:34.103806 12315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:34.103885 12310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f938dd9103547b9b65c162b6c013d28. Latest consensus state: current_term: 1 leader_uuid: "2f938dd9103547b9b65c162b6c013d28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f938dd9103547b9b65c162b6c013d28" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34627 } } }
I20260925 02:51:34.104269 12310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:34.111382 12315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:34.121680 12315 catalog_manager.cc:1347] Generated new cluster ID: c81e4c9fe06c4eb8b8267536f359b70c
I20260925 02:51:34.121806 12315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:34.141255 12315 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:51:34.142560 12315 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:34.152622 12315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f938dd9103547b9b65c162b6c013d28: Generated new TSK 0
I20260925 02:51:34.153469 12315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:51:34.167384 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34627
--builtin_ntp_servers=127.2.172.84:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 02:51:34.516269 12327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:34.516669 12327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:34.516857 12327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:34.526860 12327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:34.526999 12327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:34.527156 12327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:51:34.537158 12327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:34.538552 12327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:34.540473 12327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:34.556056 12335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:34.556126 12332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:34.556093 12333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:34.557272 12327 server_base.cc:1034] running on GCE node
I20260925 02:51:34.557950 12327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:34.558987 12327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:34.560199 12327 hybrid_clock.cc:648] HybridClock initialized: now 1790304694560155 us; error 31 us; skew 500 ppm
I20260925 02:51:34.560531 12327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:51:34.563019 12327 webserver.cc:460] Webserver started at http://127.2.172.65:40183/ using document root <none> and password file <none>
I20260925 02:51:34.563860 12327 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:34.563984 12327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:34.564323 12327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:34.567296 12327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "14597dba3665407196aae381909b5559"
format_stamp: "Formatted at 2026-09-25 02:51:34 on dist-test-slave-g1xh"
server_key: "ed368ca2d28e731fb3f4d7dadb950ac5"
server_key_iv: "b2f534ee0a6c97bf079b99edd6ad34ec"
server_key_version: "encryptionkey@0"
I20260925 02:51:34.568179 12327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "14597dba3665407196aae381909b5559"
format_stamp: "Formatted at 2026-09-25 02:51:34 on dist-test-slave-g1xh"
server_key: "ed368ca2d28e731fb3f4d7dadb950ac5"
server_key_iv: "b2f534ee0a6c97bf079b99edd6ad34ec"
server_key_version: "encryptionkey@0"
I20260925 02:51:34.573704 12327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:51:34.577639 12341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:34.579118 12327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:51:34.579286 12327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "14597dba3665407196aae381909b5559"
format_stamp: "Formatted at 2026-09-25 02:51:34 on dist-test-slave-g1xh"
server_key: "ed368ca2d28e731fb3f4d7dadb950ac5"
server_key_iv: "b2f534ee0a6c97bf079b99edd6ad34ec"
server_key_version: "encryptionkey@0"
I20260925 02:51:34.579489 12327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:34.607314 12327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:34.608266 12327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:34.608537 12327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:34.609568 12327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:34.611034 12327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:34.611121 12327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:34.611209 12327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:34.611260 12327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:34.644552 12327 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:41285
I20260925 02:51:34.644613 12453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:41285 every 8 connection(s)
I20260925 02:51:34.645931 12327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:51:34.653209 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 12327
I20260925 02:51:34.653429 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:51:34.654038 2737 external_mini_cluster.cc:1428] Setting key c71ca688f8a4593599defdf0f1bf20ef
I20260925 02:51:34.656700 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34627
--builtin_ntp_servers=127.2.172.84:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 02:51:34.659276 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:34.648714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:50439 (local address 127.2.172.126:34627)
0925 02:51:34.649169 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:34.649174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:34.650434 (+ 1260us) server_negotiation.cc:407] Connection header received
0925 02:51:34.652456 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:34.652462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:34.652552 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:34.652678 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:34.653889 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:34.654693 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:34.656415 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:34.656863 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:34.656982 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:34.658410 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:34.658415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:34.658419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:34.658437 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:34.658479 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:34.658491 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:34.658583 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:34.658813 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 02:51:34.658911 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":227,"threads_started":1}
I20260925 02:51:34.660369 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:34.649296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:34627 (local address 127.2.172.65:50439)
0925 02:51:34.650269 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:34.650309 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:34.652182 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:34.652940 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:34.652949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:34.653042 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:34.653717 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:34.653736 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:34.654929 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:34.654933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:34.656269 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:34.656282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:34.657118 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:34.657131 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:34.657353 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:34.658066 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:51:34.658069 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:34.658117 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:51:34.658203 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:34.658582 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:34.658596 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:34.658606 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 02:51:34.658808 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 02:51:34.659227 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":274,"threads_started":1}
I20260925 02:51:34.662002 12454 heartbeater.cc:347] Connected to a master server at 127.2.172.126:34627
I20260925 02:51:34.662313 12454 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:34.663064 12454 heartbeater.cc:511] Master 127.2.172.126:34627 requested a full tablet report, sending...
I20260925 02:51:34.665306 12268 ts_manager.cc:194] Registered new tserver with Master: 14597dba3665407196aae381909b5559 (127.2.172.65:41285)
I20260925 02:51:34.667510 12268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.172.65:50439
W20260925 02:51:34.998934 12458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:34.999253 12458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:34.999343 12458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:35.008081 12458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:35.008221 12458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:35.008329 12458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:51:35.018195 12458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:35.019654 12458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:35.021481 12458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:35.035501 12463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:35.036307 12458 server_base.cc:1034] running on GCE node
W20260925 02:51:35.040042 12466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:35.040364 12464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:35.041098 12458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:35.042130 12458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:35.043372 12458 hybrid_clock.cc:648] HybridClock initialized: now 1790304695043327 us; error 41 us; skew 500 ppm
I20260925 02:51:35.043752 12458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:51:35.046170 12458 webserver.cc:460] Webserver started at http://127.2.172.66:44883/ using document root <none> and password file <none>
I20260925 02:51:35.046934 12458 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:35.047032 12458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:35.047380 12458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:35.050385 12458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "17b6489a797e4dd8b6193c4a1228867b"
format_stamp: "Formatted at 2026-09-25 02:51:35 on dist-test-slave-g1xh"
server_key: "d85cc97441ed95d0d658ad1f601f3690"
server_key_iv: "38a7752c580bd19ebeccfa26385c201e"
server_key_version: "encryptionkey@0"
I20260925 02:51:35.051277 12458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "17b6489a797e4dd8b6193c4a1228867b"
format_stamp: "Formatted at 2026-09-25 02:51:35 on dist-test-slave-g1xh"
server_key: "d85cc97441ed95d0d658ad1f601f3690"
server_key_iv: "38a7752c580bd19ebeccfa26385c201e"
server_key_version: "encryptionkey@0"
I20260925 02:51:35.056726 12458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 02:51:35.060549 12472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:35.062107 12458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:51:35.062284 12458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "17b6489a797e4dd8b6193c4a1228867b"
format_stamp: "Formatted at 2026-09-25 02:51:35 on dist-test-slave-g1xh"
server_key: "d85cc97441ed95d0d658ad1f601f3690"
server_key_iv: "38a7752c580bd19ebeccfa26385c201e"
server_key_version: "encryptionkey@0"
I20260925 02:51:35.062525 12458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:35.083514 12458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:35.084560 12458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:35.084888 12458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:35.085916 12458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:35.087702 12458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:35.087800 12458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:35.087958 12458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:35.088029 12458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:35.122192 12458 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:44981
I20260925 02:51:35.122303 12584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:44981 every 8 connection(s)
I20260925 02:51:35.123652 12458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:51:35.133054 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 12458
I20260925 02:51:35.133312 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:51:35.133865 2737 external_mini_cluster.cc:1428] Setting key f276e35e6bc7bffafc7287354a351cba
I20260925 02:51:35.136781 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:35.126310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:42883 (local address 127.2.172.126:34627)
0925 02:51:35.126523 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:35.126530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:35.128045 (+ 1515us) server_negotiation.cc:407] Connection header received
0925 02:51:35.129803 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:35.129808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:35.129879 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:35.129975 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:35.131055 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.131859 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.133548 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.134181 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.134318 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:35.135832 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:35.135836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:35.135839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:35.135856 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:35.135906 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:35.135924 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:35.136011 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:35.136310 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 02:51:35.136415 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:51:35.137173 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34627
--builtin_ntp_servers=127.2.172.84:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 02:51:35.137766 12587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:35.126902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:34627 (local address 127.2.172.66:42883)
0925 02:51:35.127888 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:35.127929 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:35.129538 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:35.130219 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:35.130228 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:35.130320 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:35.130900 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:35.130916 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.132005 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.132009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:35.133382 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:35.133400 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.134394 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.134400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:35.134723 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:35.135460 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:51:35.135462 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:35.135504 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:51:35.135637 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:35.136035 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:35.136045 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:35.136051 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:51:35.136235 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 02:51:35.136550 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":282,"threads_started":1}
I20260925 02:51:35.139245 12585 heartbeater.cc:347] Connected to a master server at 127.2.172.126:34627
I20260925 02:51:35.139616 12585 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:35.140585 12585 heartbeater.cc:511] Master 127.2.172.126:34627 requested a full tablet report, sending...
I20260925 02:51:35.142210 12268 ts_manager.cc:194] Registered new tserver with Master: 17b6489a797e4dd8b6193c4a1228867b (127.2.172.66:44981)
I20260925 02:51:35.143060 12268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.172.66:42883
W20260925 02:51:35.482175 12588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:35.482476 12588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:35.482577 12588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:35.490206 12588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:35.490334 12588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:35.490447 12588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:51:35.499922 12588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:35.501421 12588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:35.503294 12588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:35.516994 12593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:35.516963 12596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:35.516896 12594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:35.517937 12588 server_base.cc:1034] running on GCE node
I20260925 02:51:35.518615 12588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:35.519735 12588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:35.520994 12588 hybrid_clock.cc:648] HybridClock initialized: now 1790304695520931 us; error 51 us; skew 500 ppm
I20260925 02:51:35.521375 12588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:51:35.524120 12588 webserver.cc:460] Webserver started at http://127.2.172.67:38429/ using document root <none> and password file <none>
I20260925 02:51:35.524911 12588 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:35.525007 12588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:35.525301 12588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:35.528405 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "f2c111701ea6463f812dbd872c40af8f"
format_stamp: "Formatted at 2026-09-25 02:51:35 on dist-test-slave-g1xh"
server_key: "ab4706ba8360fcba92bf4dbf7384027d"
server_key_iv: "f3260f78a7f10349b17a65d29c6b80f1"
server_key_version: "encryptionkey@0"
I20260925 02:51:35.529224 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "f2c111701ea6463f812dbd872c40af8f"
format_stamp: "Formatted at 2026-09-25 02:51:35 on dist-test-slave-g1xh"
server_key: "ab4706ba8360fcba92bf4dbf7384027d"
server_key_iv: "f3260f78a7f10349b17a65d29c6b80f1"
server_key_version: "encryptionkey@0"
I20260925 02:51:35.535018 12588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:51:35.539458 12602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:35.541215 12588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:51:35.541417 12588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "f2c111701ea6463f812dbd872c40af8f"
format_stamp: "Formatted at 2026-09-25 02:51:35 on dist-test-slave-g1xh"
server_key: "ab4706ba8360fcba92bf4dbf7384027d"
server_key_iv: "f3260f78a7f10349b17a65d29c6b80f1"
server_key_version: "encryptionkey@0"
I20260925 02:51:35.541683 12588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:35.581203 12588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:35.582156 12588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:35.582476 12588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:35.583585 12588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:35.585243 12588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:35.585358 12588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:35.585471 12588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:35.585573 12588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:35.619776 12588 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:38879
I20260925 02:51:35.619832 12714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:38879 every 8 connection(s)
I20260925 02:51:35.621115 12588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:51:35.624351 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 12588
I20260925 02:51:35.624547 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:51:35.625164 2737 external_mini_cluster.cc:1428] Setting key 816d2c90a94ad690b895679559ae2857
I20260925 02:51:35.634865 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:35.623839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:57081 (local address 127.2.172.126:34627)
0925 02:51:35.624056 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:35.624061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:35.625930 (+ 1869us) server_negotiation.cc:407] Connection header received
0925 02:51:35.628382 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:35.628389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:35.628487 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:35.628581 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:35.629662 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.630434 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.632323 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.632738 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.632833 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:35.634167 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:35.634172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:35.634175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:35.634195 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:35.634226 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:35.634238 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:35.634312 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:35.634524 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 02:51:35.634603 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:51:35.636191 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:35.624609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:34627 (local address 127.2.172.67:57081)
0925 02:51:35.625725 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:35.625775 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:35.628046 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:35.628787 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:35.628794 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:35.628880 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:35.629509 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:35.629525 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.630598 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.630603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:35.632176 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:35.632190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.632945 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.632950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:35.633165 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:35.633828 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:51:35.633831 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:35.633873 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:51:35.633964 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:35.634327 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:35.634335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:35.634341 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:51:35.634502 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 02:51:35.634825 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":298,"threads_started":1}
I20260925 02:51:35.637689 12715 heartbeater.cc:347] Connected to a master server at 127.2.172.126:34627
I20260925 02:51:35.637971 12715 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:35.638827 12715 heartbeater.cc:511] Master 127.2.172.126:34627 requested a full tablet report, sending...
I20260925 02:51:35.640450 12268 ts_manager.cc:194] Registered new tserver with Master: f2c111701ea6463f812dbd872c40af8f (127.2.172.67:38879)
I20260925 02:51:35.641055 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:51:35.641479 12268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.172.67:57081
W20260925 02:51:35.648005 12456 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:35.646107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50316 (local address 127.2.172.126:34627)
0925 02:51:35.646368 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:35.646376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:35.646907 (+ 531us) server_negotiation.cc:407] Connection header received
0925 02:51:35.647072 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:35.647079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:35.647181 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:35.647303 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:35.647751 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50316: BlockingRecv error: recv got EOF from 127.0.0.1:50316 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:51:35.648351 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 12327
I20260925 02:51:35.656900 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:35.649731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.2.172.65:41285)
0925 02:51:35.650293 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:35.650303 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:35.650337 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:51:35.650411 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:35.650420 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:35.650685 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:35.650869 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:35.651196 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.652362 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.653635 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.654230 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.654341 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:35.655392 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:35.655401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:35.655415 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:35.656087 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:35.656136 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:35.656157 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:35.656246 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:35.656386 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 02:51:35.656523 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":251,"threads_started":1}
I20260925 02:51:35.658145 12449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:35.741494 12454 heartbeater.cc:503] Master 127.2.172.126:34627 was elected leader, sending a full tablet report...
I20260925 02:51:35.743256 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 12327
I20260925 02:51:35.761739 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 12458
I20260925 02:51:35.769765 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:35.762784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.2.172.66:44981)
0925 02:51:35.763283 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:35.763291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:35.763317 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:51:35.763376 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:35.763382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:35.763673 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:35.763838 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:35.764231 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.765216 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.766430 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.766994 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.767153 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:35.768296 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:35.768304 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:35.768317 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:35.768936 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:35.768983 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:35.769002 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:35.769135 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:35.769286 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 02:51:35.769414 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":244,"threads_started":1}
I20260925 02:51:35.771041 12580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:35.856213 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 12458
I20260925 02:51:35.874398 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 12588
I20260925 02:51:35.883221 12728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:35.875452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32774 (local address 127.2.172.67:38879)
0925 02:51:35.876119 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:35.876126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:35.876186 (+ 60us) server_negotiation.cc:407] Connection header received
0925 02:51:35.876248 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:35.876255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:35.876480 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:35.876606 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:35.877031 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.878090 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.879442 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:35.880556 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:35.880664 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:35.881768 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:35.881777 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:35.881792 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:35.882424 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:35.882472 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:35.882492 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:35.882581 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:35.882761 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 02:51:35.882882 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":372,"threads_started":1}
I20260925 02:51:35.884660 12710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:35.969499 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 12588
I20260925 02:51:35.987972 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 12237
I20260925 02:51:35.989298 12298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:36.095158 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 12237
2026-09-25T02:51:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2545 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T02:51:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:51:36Z Disabled control of system clock
I20260925 02:51:36.171259 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:38345
--webserver_interface=127.2.172.126
--webserver_port=0
--builtin_ntp_servers=127.2.172.84:43233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:38345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:51:36.508956 12734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:36.509280 12734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:36.509429 12734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:36.517050 12734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:36.517168 12734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:36.517246 12734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:36.517354 12734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:36.517431 12734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:36.527269 12734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:43233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:38345
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:38345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:36.528653 12734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:36.530483 12734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:36.543190 12740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:36.543277 12742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:36.543298 12739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:36.544087 12734 server_base.cc:1034] running on GCE node
I20260925 02:51:36.544883 12734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:36.546722 12734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:36.547966 12734 hybrid_clock.cc:648] HybridClock initialized: now 1790304696547918 us; error 34 us; skew 500 ppm
I20260925 02:51:36.548377 12734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:51:36.550954 12734 webserver.cc:460] Webserver started at http://127.2.172.126:33055/ using document root <none> and password file <none>
I20260925 02:51:36.551805 12734 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:36.551906 12734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:36.552238 12734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:36.555371 12734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "fa081416a8ab42a8842448a0d6acac31"
format_stamp: "Formatted at 2026-09-25 02:51:36 on dist-test-slave-g1xh"
server_key: "709d936d73bd0d69610fc13101644034"
server_key_iv: "47346aa45f4e229a4a738c486c553f2c"
server_key_version: "encryptionkey@0"
I20260925 02:51:36.556326 12734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "fa081416a8ab42a8842448a0d6acac31"
format_stamp: "Formatted at 2026-09-25 02:51:36 on dist-test-slave-g1xh"
server_key: "709d936d73bd0d69610fc13101644034"
server_key_iv: "47346aa45f4e229a4a738c486c553f2c"
server_key_version: "encryptionkey@0"
I20260925 02:51:36.561807 12734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 02:51:36.565464 12748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:36.566948 12734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:51:36.567142 12734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "fa081416a8ab42a8842448a0d6acac31"
format_stamp: "Formatted at 2026-09-25 02:51:36 on dist-test-slave-g1xh"
server_key: "709d936d73bd0d69610fc13101644034"
server_key_iv: "47346aa45f4e229a4a738c486c553f2c"
server_key_version: "encryptionkey@0"
I20260925 02:51:36.567386 12734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:36.573235 12734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:36.574052 12734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:36.574381 12734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:36.594374 12734 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:38345
I20260925 02:51:36.594421 12799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:38345 every 8 connection(s)
I20260925 02:51:36.595885 12734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:51:36.597251 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 12734
I20260925 02:51:36.597469 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:51:36.598104 2737 external_mini_cluster.cc:1428] Setting key 5ab7b947599727434b25eb1b2b4e6a1e
I20260925 02:51:36.600425 12800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:36.606424 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:36.600193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.2.172.126:38345)
0925 02:51:36.601222 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:36.601234 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:36.601274 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:51:36.602839 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:36.602860 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:36.603265 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:36.603688 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:36.604751 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:36.604756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:36.604768 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:36.604780 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:36.604852 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:36.604871 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:36.605111 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:36.605287 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 02:51:36.605522 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":382,"threads_started":1}
I20260925 02:51:36.612944 12800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31: Bootstrap starting.
I20260925 02:51:36.616082 12800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:36.617194 12800 log.cc:824] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:36.620411 12800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31: No bootstrap required, opened a new log
I20260925 02:51:36.628546 12800 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa081416a8ab42a8842448a0d6acac31" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38345 } }
I20260925 02:51:36.629031 12800 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:36.629133 12800 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa081416a8ab42a8842448a0d6acac31, State: Initialized, Role: FOLLOWER
I20260925 02:51:36.629770 12800 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa081416a8ab42a8842448a0d6acac31" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38345 } }
I20260925 02:51:36.630012 12800 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:51:36.630142 12800 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:51:36.630322 12800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:36.632620 12800 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa081416a8ab42a8842448a0d6acac31" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38345 } }
I20260925 02:51:36.633155 12800 leader_election.cc:302] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa081416a8ab42a8842448a0d6acac31; no voters:
I20260925 02:51:36.633843 12800 leader_election.cc:288] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:51:36.634008 12805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:36.634797 12805 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [term 1 LEADER]: Becoming Leader. State: Replica: fa081416a8ab42a8842448a0d6acac31, State: Running, Role: LEADER
I20260925 02:51:36.635435 12805 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa081416a8ab42a8842448a0d6acac31" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38345 } }
I20260925 02:51:36.636272 12800 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:51:36.639667 12806 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa081416a8ab42a8842448a0d6acac31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa081416a8ab42a8842448a0d6acac31" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38345 } } }
I20260925 02:51:36.640205 12806 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:36.640729 12807 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa081416a8ab42a8842448a0d6acac31. Latest consensus state: current_term: 1 leader_uuid: "fa081416a8ab42a8842448a0d6acac31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa081416a8ab42a8842448a0d6acac31" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 38345 } } }
I20260925 02:51:36.641189 12807 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:36.641356 12815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:36.647012 12815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:36.656445 12815 catalog_manager.cc:1347] Generated new cluster ID: 3887ee5e4a1b407292a16d7a9623b840
I20260925 02:51:36.656571 12815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:36.675616 12815 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:51:36.676126 12815 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:36.695407 12815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa081416a8ab42a8842448a0d6acac31: Generated new TSK 0
I20260925 02:51:36.696414 12815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:51:36.703845 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:38345
--builtin_ntp_servers=127.2.172.84:43233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:51:37.075310 12824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:37.075698 12824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:37.075851 12824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:37.083699 12824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:37.083814 12824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:37.083985 12824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:51:37.093607 12824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:43233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:38345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:37.095044 12824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:37.096993 12824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:37.111786 12829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:37.111831 12833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:37.111804 12830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:37.112164 12824 server_base.cc:1034] running on GCE node
I20260925 02:51:37.113123 12824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:37.114212 12824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:37.115466 12824 hybrid_clock.cc:648] HybridClock initialized: now 1790304697115409 us; error 47 us; skew 500 ppm
I20260925 02:51:37.115849 12824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:51:37.118320 12824 webserver.cc:460] Webserver started at http://127.2.172.65:39195/ using document root <none> and password file <none>
I20260925 02:51:37.119078 12824 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:37.119225 12824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:37.119607 12824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:37.122581 12824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "519b033fa42d4660b909f31eb027faad"
format_stamp: "Formatted at 2026-09-25 02:51:37 on dist-test-slave-g1xh"
server_key: "f0dca6d19600a10572ca8ee743e9f4e5"
server_key_iv: "e4aa76a6c08cccaa4c114b0236a4c6bb"
server_key_version: "encryptionkey@0"
I20260925 02:51:37.123469 12824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "519b033fa42d4660b909f31eb027faad"
format_stamp: "Formatted at 2026-09-25 02:51:37 on dist-test-slave-g1xh"
server_key: "f0dca6d19600a10572ca8ee743e9f4e5"
server_key_iv: "e4aa76a6c08cccaa4c114b0236a4c6bb"
server_key_version: "encryptionkey@0"
I20260925 02:51:37.128980 12824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 02:51:37.132982 12839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:37.134411 12824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:51:37.134593 12824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "519b033fa42d4660b909f31eb027faad"
format_stamp: "Formatted at 2026-09-25 02:51:37 on dist-test-slave-g1xh"
server_key: "f0dca6d19600a10572ca8ee743e9f4e5"
server_key_iv: "e4aa76a6c08cccaa4c114b0236a4c6bb"
server_key_version: "encryptionkey@0"
I20260925 02:51:37.134833 12824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:37.141049 12824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:37.141943 12824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:37.142220 12824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:37.143261 12824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:37.144855 12824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:37.144981 12824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:37.145107 12824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:37.145192 12824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:37.179739 12824 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:45239
I20260925 02:51:37.179796 12952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:45239 every 8 connection(s)
I20260925 02:51:37.181106 12824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:51:37.184022 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 12824
I20260925 02:51:37.184255 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:51:37.185086 2737 external_mini_cluster.cc:1428] Setting key daf68cfbbc2a8b2f58e0a4cd69c3decf
I20260925 02:51:37.188207 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:38345
--builtin_ntp_servers=127.2.172.84:43233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 02:51:37.190749 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:37.184042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:51951 (local address 127.2.172.126:38345)
0925 02:51:37.184561 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:37.184566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:37.186146 (+ 1580us) server_negotiation.cc:407] Connection header received
0925 02:51:37.188343 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:37.188348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:37.188419 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:37.188555 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:37.189947 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:37.189953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:37.189958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:37.189984 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:37.190036 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:37.190059 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:37.190192 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:37.190384 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 02:51:37.190452 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":322,"threads_started":1}
I20260925 02:51:37.191783 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:37.184812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38345 (local address 127.2.172.65:51951)
0925 02:51:37.185911 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:37.185963 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:37.188081 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:37.188817 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:37.188826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:37.188934 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:37.189661 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:51:37.189664 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:37.189700 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:51:37.189783 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:37.190161 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:37.190174 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:37.190181 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:51:37.190486 (+ 305us) client_negotiation.cc:240] Negotiation successful
0925 02:51:37.190750 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":361,"threads_started":1}
I20260925 02:51:37.193418 12953 heartbeater.cc:347] Connected to a master server at 127.2.172.126:38345
I20260925 02:51:37.193780 12953 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:37.194383 12953 heartbeater.cc:511] Master 127.2.172.126:38345 requested a full tablet report, sending...
I20260925 02:51:37.196730 12765 ts_manager.cc:194] Registered new tserver with Master: 519b033fa42d4660b909f31eb027faad (127.2.172.65:45239)
W20260925 02:51:37.534837 12957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:37.535172 12957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:37.535336 12957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:37.544572 12957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:37.544694 12957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:37.544936 12957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:51:37.561367 12957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:43233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:38345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:37.562824 12957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:37.564949 12957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:37.579859 12965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:37.579895 12962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:37.579793 12963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:37.581468 12957 server_base.cc:1034] running on GCE node
I20260925 02:51:37.582283 12957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:37.583438 12957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:37.584779 12957 hybrid_clock.cc:648] HybridClock initialized: now 1790304697584722 us; error 52 us; skew 500 ppm
I20260925 02:51:37.585186 12957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:51:37.588065 12957 webserver.cc:460] Webserver started at http://127.2.172.66:35561/ using document root <none> and password file <none>
I20260925 02:51:37.588860 12957 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:37.589005 12957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:37.589368 12957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:37.592597 12957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "177b7fffccd544aaa7ff4d1ecb99ee8e"
format_stamp: "Formatted at 2026-09-25 02:51:37 on dist-test-slave-g1xh"
server_key: "9a2b2cac0536df67bb36bc66f65775d4"
server_key_iv: "2f44be16d30502d58f47ac14c3df1a14"
server_key_version: "encryptionkey@0"
I20260925 02:51:37.593531 12957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "177b7fffccd544aaa7ff4d1ecb99ee8e"
format_stamp: "Formatted at 2026-09-25 02:51:37 on dist-test-slave-g1xh"
server_key: "9a2b2cac0536df67bb36bc66f65775d4"
server_key_iv: "2f44be16d30502d58f47ac14c3df1a14"
server_key_version: "encryptionkey@0"
I20260925 02:51:37.599402 12957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:51:37.603370 12971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:37.605034 12957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:51:37.605238 12957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "177b7fffccd544aaa7ff4d1ecb99ee8e"
format_stamp: "Formatted at 2026-09-25 02:51:37 on dist-test-slave-g1xh"
server_key: "9a2b2cac0536df67bb36bc66f65775d4"
server_key_iv: "2f44be16d30502d58f47ac14c3df1a14"
server_key_version: "encryptionkey@0"
I20260925 02:51:37.605479 12957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:37.611536 12957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:37.612495 12957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:37.612794 12957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:37.613799 12957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:37.615279 12957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:37.615404 12957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:37.615643 12957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:37.615738 12957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 02:51:37.650312 12957 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:37389
I20260925 02:51:37.650377 13084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:37389 every 8 connection(s)
I20260925 02:51:37.651825 12957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:51:37.657907 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 12957
I20260925 02:51:37.658093 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:51:37.658574 2737 external_mini_cluster.cc:1428] Setting key b00106862f1cf54d911c964cdc7d5ffe
I20260925 02:51:37.660563 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:37.654556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:40245 (local address 127.2.172.126:38345)
0925 02:51:37.654751 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:37.654755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:37.656170 (+ 1415us) server_negotiation.cc:407] Connection header received
0925 02:51:37.657948 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:37.657953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:37.658071 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:37.658198 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:37.659749 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:37.659755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:37.659759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:37.659790 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:37.659824 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:37.659837 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:37.659934 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:37.660285 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 02:51:37.660336 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:51:37.661722 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:37.655056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38345 (local address 127.2.172.66:40245)
0925 02:51:37.656007 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:37.656043 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:37.657667 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:37.658475 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:37.658484 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:37.658582 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:37.659342 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:51:37.659345 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:37.659396 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:51:37.659522 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:37.659953 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:37.659963 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:37.659971 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:51:37.660364 (+ 393us) client_negotiation.cc:240] Negotiation successful
0925 02:51:37.660709 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":272,"threads_started":1}
I20260925 02:51:37.661880 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:38345
--builtin_ntp_servers=127.2.172.84:43233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 02:51:37.663344 13085 heartbeater.cc:347] Connected to a master server at 127.2.172.126:38345
I20260925 02:51:37.663717 13085 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:37.664358 13085 heartbeater.cc:511] Master 127.2.172.126:38345 requested a full tablet report, sending...
I20260925 02:51:37.666011 12765 ts_manager.cc:194] Registered new tserver with Master: 177b7fffccd544aaa7ff4d1ecb99ee8e (127.2.172.66:37389)
W20260925 02:51:38.009692 13088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:38.009979 13088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:38.010066 13088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:38.017536 13088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:38.017661 13088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:38.017779 13088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:51:38.027176 13088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:43233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:38345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:38.028545 13088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:38.030288 13088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:38.044047 13094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:38.044098 13096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:38.044020 13093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:38.044803 13088 server_base.cc:1034] running on GCE node
I20260925 02:51:38.045418 13088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:38.046429 13088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:38.047660 13088 hybrid_clock.cc:648] HybridClock initialized: now 1790304698047592 us; error 63 us; skew 500 ppm
I20260925 02:51:38.047991 13088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:51:38.050372 13088 webserver.cc:460] Webserver started at http://127.2.172.67:33013/ using document root <none> and password file <none>
I20260925 02:51:38.051095 13088 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:38.051213 13088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:38.051620 13088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:38.054590 13088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "485ef125c7eb4caf95a48b465a73026a"
format_stamp: "Formatted at 2026-09-25 02:51:38 on dist-test-slave-g1xh"
server_key: "c25a7323a0714346a2642e3c0388a15f"
server_key_iv: "cfd89a583b357effa0d64391af62273c"
server_key_version: "encryptionkey@0"
I20260925 02:51:38.055449 13088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "485ef125c7eb4caf95a48b465a73026a"
format_stamp: "Formatted at 2026-09-25 02:51:38 on dist-test-slave-g1xh"
server_key: "c25a7323a0714346a2642e3c0388a15f"
server_key_iv: "cfd89a583b357effa0d64391af62273c"
server_key_version: "encryptionkey@0"
I20260925 02:51:38.061008 13088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:51:38.064800 13103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:38.066282 13088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:51:38.066478 13088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "485ef125c7eb4caf95a48b465a73026a"
format_stamp: "Formatted at 2026-09-25 02:51:38 on dist-test-slave-g1xh"
server_key: "c25a7323a0714346a2642e3c0388a15f"
server_key_iv: "cfd89a583b357effa0d64391af62273c"
server_key_version: "encryptionkey@0"
I20260925 02:51:38.066727 13088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:38.072798 13088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:38.073675 13088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:38.073966 13088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:38.074967 13088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:38.076527 13088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:38.076648 13088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:38.076790 13088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:38.076877 13088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:38.112267 13088 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:39721
I20260925 02:51:38.112321 13215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:39721 every 8 connection(s)
I20260925 02:51:38.113644 13088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:51:38.118438 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 13088
I20260925 02:51:38.118625 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:51:38.119257 2737 external_mini_cluster.cc:1428] Setting key e87059098a5b696c884e041629a28b75
I20260925 02:51:38.123255 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:38.116532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:42115 (local address 127.2.172.126:38345)
0925 02:51:38.116738 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:38.116743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:38.118210 (+ 1467us) server_negotiation.cc:407] Connection header received
0925 02:51:38.120729 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:38.120734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:38.120825 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:38.120941 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:38.122431 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:38.122445 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:38.122450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:38.122467 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:38.122502 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:38.122515 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:38.122656 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:38.122866 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 02:51:38.122941 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:51:38.124439 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:38.116940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:38345 (local address 127.2.172.67:42115)
0925 02:51:38.117964 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:38.118015 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:38.120375 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:38.121199 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:38.121208 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:38.121308 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:38.122036 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:51:38.122039 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:38.122096 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:51:38.122239 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:38.122618 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:38.122629 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:38.122638 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:51:38.122965 (+ 327us) client_negotiation.cc:240] Negotiation successful
0925 02:51:38.123282 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":292,"threads_started":1}
I20260925 02:51:38.125823 13216 heartbeater.cc:347] Connected to a master server at 127.2.172.126:38345
I20260925 02:51:38.126109 13216 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:38.126737 13216 heartbeater.cc:511] Master 127.2.172.126:38345 requested a full tablet report, sending...
I20260925 02:51:38.128389 12765 ts_manager.cc:194] Registered new tserver with Master: 485ef125c7eb4caf95a48b465a73026a (127.2.172.67:39721)
I20260925 02:51:38.135732 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:51:38.142649 12955 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:38.140965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.2.172.126:38345)
0925 02:51:38.141147 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:38.141151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:38.141788 (+ 637us) server_negotiation.cc:407] Connection header received
0925 02:51:38.141866 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:38.141870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:38.141941 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:38.142030 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:38.142316 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52534: BlockingRecv error: recv got EOF from 127.0.0.1:52534 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:51:38.143642 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 12824
I20260925 02:51:38.147665 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:38.144880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.2.172.65:45239)
0925 02:51:38.145462 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:38.145469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:38.145498 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:51:38.145566 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:38.145572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:38.145851 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:38.145991 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:38.146923 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:38.146930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:38.146941 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:38.146959 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:38.146998 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:38.147012 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:38.147118 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:38.147244 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 02:51:38.147328 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":310,"threads_started":1}
I20260925 02:51:38.148976 12948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:38.231841 12953 heartbeater.cc:503] Master 127.2.172.126:38345 was elected leader, sending a full tablet report...
I20260925 02:51:38.233161 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 12824
W20260925 02:51:38.251060 12751 connection.cc:535] server connection from 127.2.172.65:51951 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:51:38.251257 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 12957
I20260925 02:51:38.254956 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:38.252300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.2.172.66:37389)
0925 02:51:38.252900 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:38.252908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:38.252937 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:51:38.253005 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:38.253011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:38.253232 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:38.253368 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:38.254332 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:38.254338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:38.254350 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:38.254370 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:38.254415 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:38.254429 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:38.254560 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:38.254617 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 02:51:38.254698 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":310,"threads_started":1}
I20260925 02:51:38.256457 13080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:38.343007 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 12957
I20260925 02:51:38.360921 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 13088
I20260925 02:51:38.365233 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:38.361999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.2.172.67:39721)
0925 02:51:38.362671 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:38.362678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:38.362709 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:51:38.362774 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:38.362781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:38.362996 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:38.363182 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:38.364434 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:38.364441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:38.364452 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:38.364468 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:38.364506 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:38.364520 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:38.364727 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:38.364805 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 02:51:38.364909 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":358,"threads_started":1}
I20260925 02:51:38.366578 13211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:38.452142 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 13088
I20260925 02:51:38.470103 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 12734
I20260925 02:51:38.471391 12795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:38.575812 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 12734
2026-09-25T02:51:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2484 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:51:38 dist-test-slave-g1xh krb5kdc[13234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:51:38 dist-test-slave-g1xh krb5kdc[13234](info): set up 2 sockets
Sep 25 02:51:38 dist-test-slave-g1xh krb5kdc[13234](info): commencing operation
krb5kdc: starting...
W20260925 02:51:39.691140 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:51:39 dist-test-slave-g1xh krb5kdc[13234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:51:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:51:39.955209 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:41507
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:41507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:40.295243 13247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:40.295658 13247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:40.295832 13247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:40.304001 13247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:40.304181 13247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:40.304302 13247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:40.304404 13247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:40.304489 13247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:40.315672 13247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:41507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:40.317164 13247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:40.318598 13247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:40.331105 13255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:40.331038 13252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:40.331133 13253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:40.332294 13247 server_base.cc:1034] running on GCE node
I20260925 02:51:40.333125 13247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:40.334610 13247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:40.335870 13247 hybrid_clock.cc:648] HybridClock initialized: now 1790304700335815 us; error 44 us; skew 500 ppm
I20260925 02:51:40.336274 13247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:40 dist-test-slave-g1xh krb5kdc[13234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304700, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:40.343968 13247 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:51:40.345760 13247 webserver.cc:460] Webserver started at http://127.2.172.126:36399/ using document root <none> and password file <none>
I20260925 02:51:40.346503 13247 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:40.346668 13247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:40.347026 13247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:40.350404 13247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "7c9703a490d54cc2b69667049b580a71"
format_stamp: "Formatted at 2026-09-25 02:51:40 on dist-test-slave-g1xh"
server_key: "779627593fab66c54c3454d7d294acaf"
server_key_iv: "497c763b0b74f4f60c87069fa54c4fd5"
server_key_version: "encryptionkey@0"
I20260925 02:51:40.351250 13247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "7c9703a490d54cc2b69667049b580a71"
format_stamp: "Formatted at 2026-09-25 02:51:40 on dist-test-slave-g1xh"
server_key: "779627593fab66c54c3454d7d294acaf"
server_key_iv: "497c763b0b74f4f60c87069fa54c4fd5"
server_key_version: "encryptionkey@0"
I20260925 02:51:40.357141 13247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 02:51:40.361057 13262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:40.362468 13247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:51:40.362710 13247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "7c9703a490d54cc2b69667049b580a71"
format_stamp: "Formatted at 2026-09-25 02:51:40 on dist-test-slave-g1xh"
server_key: "779627593fab66c54c3454d7d294acaf"
server_key_iv: "497c763b0b74f4f60c87069fa54c4fd5"
server_key_version: "encryptionkey@0"
I20260925 02:51:40.362932 13247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:40.379241 13247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:40.384158 13247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:40.384485 13247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:40.405542 13247 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:41507
I20260925 02:51:40.405547 13314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:41507 every 8 connection(s)
I20260925 02:51:40.407122 13247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:51:40.411267 13315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:40.412027 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 13247
I20260925 02:51:40.412267 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:51:40.412940 2737 external_mini_cluster.cc:1428] Setting key 5dbc0d7315814cef661e7efdf8be8685
I20260925 02:51:40.427006 13315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71: Bootstrap starting.
Sep 25 02:51:40 dist-test-slave-g1xh krb5kdc[13234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:51:40.430495 13315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:40.432032 13315 log.cc:824] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:40.435968 13315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71: No bootstrap required, opened a new log
I20260925 02:51:40.439888 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:40.415224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.2.172.126:41507)
0925 02:51:40.416145 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:40.416161 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:40.416222 (+ 61us) server_negotiation.cc:407] Connection header received
0925 02:51:40.418585 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:40.418623 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:40.419271 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:40.419806 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:40.421372 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:40.423436 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:40.424894 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:40.426029 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:40.426233 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:40.430552 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:40.430565 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:40.430589 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:40.430648 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:40.434866 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:40.435789 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:40.435798 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:40.435806 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:40.436029 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:40.436467 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:40.436475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:40.436478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:40.437264 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:40.437587 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:40.437931 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 02:51:40.438417 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":408,"threads_started":1}
I20260925 02:51:40.444978 13315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9703a490d54cc2b69667049b580a71" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41507 } }
I20260925 02:51:40.445338 13315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:40.445415 13315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9703a490d54cc2b69667049b580a71, State: Initialized, Role: FOLLOWER
I20260925 02:51:40.446003 13315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9703a490d54cc2b69667049b580a71" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41507 } }
I20260925 02:51:40.446201 13315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:51:40.446300 13315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:51:40.446420 13315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:40.448693 13315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9703a490d54cc2b69667049b580a71" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41507 } }
I20260925 02:51:40.449196 13315 leader_election.cc:302] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c9703a490d54cc2b69667049b580a71; no voters:
I20260925 02:51:40.449833 13315 leader_election.cc:288] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:51:40.449992 13320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:40.450698 13320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [term 1 LEADER]: Becoming Leader. State: Replica: 7c9703a490d54cc2b69667049b580a71, State: Running, Role: LEADER
I20260925 02:51:40.451236 13320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9703a490d54cc2b69667049b580a71" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41507 } }
I20260925 02:51:40.451949 13315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:51:40.454027 13321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c9703a490d54cc2b69667049b580a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9703a490d54cc2b69667049b580a71" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41507 } } }
I20260925 02:51:40.454434 13321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:40.454408 13322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c9703a490d54cc2b69667049b580a71. Latest consensus state: current_term: 1 leader_uuid: "7c9703a490d54cc2b69667049b580a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9703a490d54cc2b69667049b580a71" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41507 } } }
I20260925 02:51:40.454792 13322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:40.456809 13326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:40.462249 13326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:40.472489 13326 catalog_manager.cc:1347] Generated new cluster ID: 9c9cb72c0e724c4a9eb418821f38fedc
I20260925 02:51:40.472628 13326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:40.484341 13326 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:51:40.486284 13326 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:40.494122 13326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71: Generated new TSK 0
I20260925 02:51:40.495213 13326 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:51:40.563448 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:41507
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:40.912022 13341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:40.912310 13341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:40.912405 13341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:40.919680 13341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:40.919795 13341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:40.919905 13341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:51:40.929854 13341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:40.931093 13341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:40.932593 13341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:40.946521 13347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:40.946538 13346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:40.946543 13349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:40.948132 13341 server_base.cc:1034] running on GCE node
I20260925 02:51:40.948812 13341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:40.949816 13341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:40.951041 13341 hybrid_clock.cc:648] HybridClock initialized: now 1790304700950991 us; error 41 us; skew 500 ppm
I20260925 02:51:40.951391 13341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:40 dist-test-slave-g1xh krb5kdc[13234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304700, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:40.955027 13341 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:51:40.956755 13341 webserver.cc:460] Webserver started at http://127.2.172.65:45121/ using document root <none> and password file <none>
I20260925 02:51:40.957486 13341 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:40.957643 13341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:40.957976 13341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:40.961100 13341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "501a953016ff4745ab448f42616afa67"
format_stamp: "Formatted at 2026-09-25 02:51:40 on dist-test-slave-g1xh"
server_key: "5438da16c76aeac5dac0d9df06e4cf0a"
server_key_iv: "9721ec2681a682f30d28f8375d046134"
server_key_version: "encryptionkey@0"
I20260925 02:51:40.961930 13341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "501a953016ff4745ab448f42616afa67"
format_stamp: "Formatted at 2026-09-25 02:51:40 on dist-test-slave-g1xh"
server_key: "5438da16c76aeac5dac0d9df06e4cf0a"
server_key_iv: "9721ec2681a682f30d28f8375d046134"
server_key_version: "encryptionkey@0"
I20260925 02:51:40.967885 13341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260925 02:51:40.971716 13356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:40.973229 13341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:51:40.973420 13341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "501a953016ff4745ab448f42616afa67"
format_stamp: "Formatted at 2026-09-25 02:51:40 on dist-test-slave-g1xh"
server_key: "5438da16c76aeac5dac0d9df06e4cf0a"
server_key_iv: "9721ec2681a682f30d28f8375d046134"
server_key_version: "encryptionkey@0"
I20260925 02:51:40.973637 13341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:40.987645 13341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:40.992471 13341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:40.992786 13341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:40.993858 13341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:40.995289 13341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:40.995396 13341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:40.995589 13341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:40.995676 13341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 02:51:41.030777 13341 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:33101
I20260925 02:51:41.030812 13469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:33101 every 8 connection(s)
I20260925 02:51:41.032111 13341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:51:41.040006 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 13341
I20260925 02:51:41.040254 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:51:41.041081 2737 external_mini_cluster.cc:1428] Setting key 7e12f03ced40c0eff0eaf3f52ccee520
Sep 25 02:51:41 dist-test-slave-g1xh krb5kdc[13234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304700, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:51:41.056500 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:41.034921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:38897 (local address 127.2.172.126:41507)
0925 02:51:41.035465 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:41.035470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:41.036280 (+ 810us) server_negotiation.cc:407] Connection header received
0925 02:51:41.038302 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:41.038308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:41.038425 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:41.038559 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:41.040662 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:41.041525 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:41.043783 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:41.044273 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:41.044402 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:41.049500 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:41.049514 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:41.049517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:41.049554 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:41.052544 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:41.053664 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:41.053669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:41.053671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:41.053809 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:41.054338 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:41.054346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:41.054352 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:41.054786 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:41.054944 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:41.055727 (+ 783us) server_negotiation.cc:299] Negotiation successful
0925 02:51:41.055999 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":309,"threads_started":1}
I20260925 02:51:41.057417 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:41.035333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:41507 (local address 127.2.172.65:38897)
0925 02:51:41.036109 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:41.036160 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:41.037966 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:41.038827 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:41.038837 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:41.039470 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:41.040449 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:41.040480 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:41.041729 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:41.041734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:41.043590 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:41.043609 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:41.044554 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:41.044565 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:41.044956 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:41.045789 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:41.045827 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:41.049195 (+ 3368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:41.052735 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:41.052748 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:41.052772 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:41.053514 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:41.053929 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:41.053934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:41.053939 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:41.054205 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:41.054921 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:41.054928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:41.055300 (+ 372us) client_negotiation.cc:769] Sending connection context
0925 02:51:41.055608 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 02:51:41.056062 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":338,"threads_started":1}
I20260925 02:51:41.059263 13470 heartbeater.cc:347] Connected to a master server at 127.2.172.126:41507
I20260925 02:51:41.059650 13470 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:41.060532 13470 heartbeater.cc:511] Master 127.2.172.126:41507 requested a full tablet report, sending...
I20260925 02:51:41.062879 13280 ts_manager.cc:194] Registered new tserver with Master: 501a953016ff4745ab448f42616afa67 (127.2.172.65:33101)
I20260925 02:51:41.065135 13280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:38897
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:51:41.103963 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:41507
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:41.447145 13476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:41.447448 13476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:41.447597 13476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:41.455251 13476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:41.455377 13476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:41.455494 13476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:51:41.465745 13476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:41.467193 13476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:41.468734 13476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:41.482976 13481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:41.482892 13482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:41.482928 13484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:41.489634 13476 server_base.cc:1034] running on GCE node
I20260925 02:51:41.490517 13476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:41.491823 13476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:41.493155 13476 hybrid_clock.cc:648] HybridClock initialized: now 1790304701493101 us; error 66 us; skew 500 ppm
I20260925 02:51:41.493566 13476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:41 dist-test-slave-g1xh krb5kdc[13234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304701, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:41.497172 13476 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:51:41.499001 13476 webserver.cc:460] Webserver started at http://127.2.172.66:35807/ using document root <none> and password file <none>
I20260925 02:51:41.499838 13476 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:41.499995 13476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:41.500314 13476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:41.503378 13476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "60ac9c66c54947f99ad745b97743cd0f"
format_stamp: "Formatted at 2026-09-25 02:51:41 on dist-test-slave-g1xh"
server_key: "899c48ca21285324046279feff1d63e9"
server_key_iv: "39286c82679e041b47d6f774d12dafaa"
server_key_version: "encryptionkey@0"
I20260925 02:51:41.504288 13476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "60ac9c66c54947f99ad745b97743cd0f"
format_stamp: "Formatted at 2026-09-25 02:51:41 on dist-test-slave-g1xh"
server_key: "899c48ca21285324046279feff1d63e9"
server_key_iv: "39286c82679e041b47d6f774d12dafaa"
server_key_version: "encryptionkey@0"
I20260925 02:51:41.510094 13476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:51:41.514040 13491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:41.515709 13476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:51:41.515908 13476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "60ac9c66c54947f99ad745b97743cd0f"
format_stamp: "Formatted at 2026-09-25 02:51:41 on dist-test-slave-g1xh"
server_key: "899c48ca21285324046279feff1d63e9"
server_key_iv: "39286c82679e041b47d6f774d12dafaa"
server_key_version: "encryptionkey@0"
I20260925 02:51:41.516153 13476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:41.532884 13476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:41.537590 13476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:41.538014 13476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:41.539135 13476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:41.540731 13476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:41.540850 13476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:41.541005 13476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:41.541079 13476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:41.589076 13476 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:33331
I20260925 02:51:41.589102 13605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:33331 every 8 connection(s)
I20260925 02:51:41.590517 13476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:51:41.600616 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 13476
I20260925 02:51:41.600826 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:51:41.601390 2737 external_mini_cluster.cc:1428] Setting key a3b662e00b02790e2e4853d4d53749c3
Sep 25 02:51:41 dist-test-slave-g1xh krb5kdc[13234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304701, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:51:41.619536 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:41.593346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:56275 (local address 127.2.172.126:41507)
0925 02:51:41.593863 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:41.593870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:41.594916 (+ 1046us) server_negotiation.cc:407] Connection header received
0925 02:51:41.596847 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:41.596852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:41.596938 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:41.597105 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:41.598723 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:41.599507 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:41.601885 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:41.602586 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:41.602727 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:41.611899 (+ 9172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:41.611905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:41.611909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:41.611949 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:41.615701 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:41.616601 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:41.616610 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:41.616616 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:41.616788 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:41.617307 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:41.617315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:41.617320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:41.617755 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:41.617944 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:41.618732 (+ 788us) server_negotiation.cc:299] Negotiation successful
0925 02:51:41.619010 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":271,"threads_started":1}
I20260925 02:51:41.620411 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:41.593934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:41507 (local address 127.2.172.66:56275)
0925 02:51:41.594731 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:41.594779 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:41.596523 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:41.597336 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:41.597345 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:41.597899 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:41.598526 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:41.598545 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:41.599742 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:41.599746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:41.601698 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:41.601715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:41.602834 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:41.602839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:41.603053 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:41.608161 (+ 5108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:41.608205 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:41.611526 (+ 3321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:41.615915 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:41.615925 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:41.615942 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:41.616437 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:41.616957 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:41.616960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:41.616963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:41.617179 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:41.617891 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:41.617898 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:41.618339 (+ 441us) client_negotiation.cc:769] Sending connection context
0925 02:51:41.618587 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 02:51:41.619031 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":295,"threads_started":1}
I20260925 02:51:41.622184 13606 heartbeater.cc:347] Connected to a master server at 127.2.172.126:41507
I20260925 02:51:41.622541 13606 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:41.623347 13606 heartbeater.cc:511] Master 127.2.172.126:41507 requested a full tablet report, sending...
I20260925 02:51:41.625065 13280 ts_manager.cc:194] Registered new tserver with Master: 60ac9c66c54947f99ad745b97743cd0f (127.2.172.66:33331)
I20260925 02:51:41.625929 13280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:56275
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:51:41.664076 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:41507
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:42.021291 13612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:42.021620 13612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:42.021795 13612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:42.029343 13612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:42.029513 13612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:42.029691 13612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:51:42.040061 13612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:51:42.041368 13612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:42.042716 13612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:42.056250 13617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:42.056222 13618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:42.056586 13620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:42.057252 13612 server_base.cc:1034] running on GCE node
I20260925 02:51:42.057947 13612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:42.058990 13612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:42.060243 13612 hybrid_clock.cc:648] HybridClock initialized: now 1790304702060173 us; error 49 us; skew 500 ppm
I20260925 02:51:42.060616 13612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:51:42 dist-test-slave-g1xh krb5kdc[13234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304702, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:42.063942 13612 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:51:42.065403 13612 webserver.cc:460] Webserver started at http://127.2.172.67:43675/ using document root <none> and password file <none>
I20260925 02:51:42.066107 13612 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:42.066256 13612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:42.066632 13612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:42.068701 13470 heartbeater.cc:503] Master 127.2.172.126:41507 was elected leader, sending a full tablet report...
I20260925 02:51:42.070039 13612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad"
format_stamp: "Formatted at 2026-09-25 02:51:42 on dist-test-slave-g1xh"
server_key: "968c6ce8ce56bcf14b39a4f92021cc0a"
server_key_iv: "949b802ca7a350b02c7fda3536c0be83"
server_key_version: "encryptionkey@0"
I20260925 02:51:42.070912 13612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad"
format_stamp: "Formatted at 2026-09-25 02:51:42 on dist-test-slave-g1xh"
server_key: "968c6ce8ce56bcf14b39a4f92021cc0a"
server_key_iv: "949b802ca7a350b02c7fda3536c0be83"
server_key_version: "encryptionkey@0"
I20260925 02:51:42.076292 13612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:51:42.080073 13627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:42.081513 13612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:51:42.081712 13612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad"
format_stamp: "Formatted at 2026-09-25 02:51:42 on dist-test-slave-g1xh"
server_key: "968c6ce8ce56bcf14b39a4f92021cc0a"
server_key_iv: "949b802ca7a350b02c7fda3536c0be83"
server_key_version: "encryptionkey@0"
I20260925 02:51:42.081902 13612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:42.106300 13612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:42.111220 13612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:42.111632 13612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:42.112690 13612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:42.114127 13612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:42.114202 13612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:42.114377 13612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:42.114430 13612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:42.148789 13612 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:44497
I20260925 02:51:42.148861 13741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:44497 every 8 connection(s)
I20260925 02:51:42.150110 13612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:51:42.151418 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 13612
I20260925 02:51:42.151644 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:51:42.152223 2737 external_mini_cluster.cc:1428] Setting key bca646c2e47c96db61138ed30a0be620
Sep 25 02:51:42 dist-test-slave-g1xh krb5kdc[13234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304702, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:51:42.171885 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.152985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:59143 (local address 127.2.172.126:41507)
0925 02:51:42.153507 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:42.153513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:42.154644 (+ 1131us) server_negotiation.cc:407] Connection header received
0925 02:51:42.156853 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:42.156857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:42.157033 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:42.157141 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:42.158899 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.159834 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.161373 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.161784 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.161891 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.165442 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:42.165447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:42.165451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:42.165484 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:42.168866 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:42.169583 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:42.169587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:42.169589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:42.169705 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:42.170101 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:42.170104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:42.170105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:42.170359 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:42.170476 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:42.171330 (+ 854us) server_negotiation.cc:299] Negotiation successful
0925 02:51:42.171506 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":247,"threads_started":1}
I20260925 02:51:42.172848 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.153599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:41507 (local address 127.2.172.67:59143)
0925 02:51:42.154432 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:42.154508 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:42.156589 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:42.157412 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:42.157421 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:42.158128 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:42.158728 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:42.158746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.159997 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.160003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:42.161235 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:42.161247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.162021 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.162029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:42.162238 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.162970 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:42.162993 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:42.165235 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:42.169000 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:42.169010 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:42.169030 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:42.169465 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:42.169809 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:42.169813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:42.169815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:42.169996 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:42.170502 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:42.170509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:42.170862 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 02:51:42.171166 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 02:51:42.171707 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":317,"threads_started":1}
I20260925 02:51:42.174316 13742 heartbeater.cc:347] Connected to a master server at 127.2.172.126:41507
I20260925 02:51:42.174635 13742 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:42.175381 13742 heartbeater.cc:511] Master 127.2.172.126:41507 requested a full tablet report, sending...
I20260925 02:51:42.176950 13280 ts_manager.cc:194] Registered new tserver with Master: c247eb4ac0d14b1ba815777fbcbcc4ad (127.2.172.67:44497)
I20260925 02:51:42.178020 13280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:59143
I20260925 02:51:42.180972 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:51:42.198450 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.185626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.2.172.126:41507)
0925 02:51:42.185874 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:42.185884 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:42.186381 (+ 497us) server_negotiation.cc:407] Connection header received
0925 02:51:42.186507 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:42.186511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:42.186573 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:42.186653 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:42.188199 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.188919 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.190159 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.190568 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.190660 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.193121 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:42.193126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:42.193130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:42.193162 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:42.195987 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:42.196691 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:42.196695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:42.196697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:42.196822 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:42.197162 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:42.197165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:42.197168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:42.197481 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:42.197660 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:42.197990 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 02:51:42.198156 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:51:42.202919 13280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:51:42.210728 13280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:51:42.238003 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.225433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:33331 (local address 127.0.0.1:43184)
0925 02:51:42.226117 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:42.226135 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:42.226250 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:42.227189 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:42.227196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:42.227237 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:42.227714 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:42.227733 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.230300 (+ 2567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.230309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:42.233569 (+ 3260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:42.233592 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.236627 (+ 3035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.236637 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:42.236923 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.237002 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 02:51:42.237170 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 02:51:42.237440 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":366,"threads_started":1}
I20260925 02:51:42.239106 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.225789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.2.172.66:33331)
0925 02:51:42.226358 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:42.226365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:42.226387 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:51:42.226470 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:42.226475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:42.226690 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:42.226905 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:42.227980 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.230062 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.233843 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.235958 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.236138 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.236419 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:42.238489 (+ 2070us) server_negotiation.cc:299] Negotiation successful
0925 02:51:42.238704 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":283,"threads_started":1}
I20260925 02:51:42.244930 13539 tablet_service.cc:1467] Processing CreateTablet for tablet 05d23aaff5df4f139abd81ea645fde7d (DEFAULT_TABLE table=test-table [id=fcda88c26ba64ed38f4a573aee00029f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:42.246171 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.224341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:33101 (local address 127.0.0.1:41854)
0925 02:51:42.225027 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:42.225078 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:42.225253 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:42.238325 (+ 13072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:42.238330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:42.238361 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:42.238777 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:42.238792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.241813 (+ 3021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.241817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:42.243543 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:42.243616 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.245399 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.245405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:42.245605 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.245669 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 02:51:42.245764 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 02:51:42.245885 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":317,"threads_started":1}
I20260925 02:51:42.246598 13539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d23aaff5df4f139abd81ea645fde7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:42.246927 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.224643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.2.172.65:33101)
0925 02:51:42.237510 (+ 12867us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:42.237523 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:42.237565 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:51:42.237710 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:42.237722 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:42.238074 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:42.238339 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:42.238961 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.240765 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.244038 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.245152 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.245314 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.245580 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:42.246482 (+ 902us) server_negotiation.cc:299] Negotiation successful
0925 02:51:42.246649 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12608,"thread_start_us":13981,"threads_started":1}
I20260925 02:51:42.249274 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.226859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:44497 (local address 127.0.0.1:54590)
0925 02:51:42.240426 (+ 13567us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:42.240451 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:42.240585 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:42.241948 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:42.241953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:42.241987 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:42.242430 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:42.242440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.244420 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.244427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:42.246652 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:42.246670 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.248507 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.248511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:42.248636 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.248660 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:51:42.248792 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 02:51:42.248929 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13373,"thread_start_us":13192,"threads_started":1}
I20260925 02:51:42.249353 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.227088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.2.172.67:44497)
0925 02:51:42.241395 (+ 14307us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:42.241402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:42.241430 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:51:42.241504 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:42.241510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:42.241706 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:42.241946 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:42.242649 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.244251 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.246878 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.248304 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.248449 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.248680 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:42.248848 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 02:51:42.249011 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14137,"thread_start_us":13967,"threads_started":1}
I20260925 02:51:42.250970 13404 tablet_service.cc:1467] Processing CreateTablet for tablet 05d23aaff5df4f139abd81ea645fde7d (DEFAULT_TABLE table=test-table [id=fcda88c26ba64ed38f4a573aee00029f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:42.252610 13404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d23aaff5df4f139abd81ea645fde7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:42.255862 13676 tablet_service.cc:1467] Processing CreateTablet for tablet 05d23aaff5df4f139abd81ea645fde7d (DEFAULT_TABLE table=test-table [id=fcda88c26ba64ed38f4a573aee00029f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:42.257360 13676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d23aaff5df4f139abd81ea645fde7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:42.262753 13761 tablet_bootstrap.cc:492] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f: Bootstrap starting.
I20260925 02:51:42.266782 13762 tablet_bootstrap.cc:492] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67: Bootstrap starting.
I20260925 02:51:42.267395 13761 tablet_bootstrap.cc:654] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:42.268772 13761 log.cc:824] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:42.269944 13762 tablet_bootstrap.cc:654] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:42.271236 13762 log.cc:824] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:42.272310 13761 tablet_bootstrap.cc:492] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f: No bootstrap required, opened a new log
I20260925 02:51:42.272677 13761 ts_tablet_manager.cc:1397] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260925 02:51:42.274235 13762 tablet_bootstrap.cc:492] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67: No bootstrap required, opened a new log
I20260925 02:51:42.274505 13762 ts_tablet_manager.cc:1397] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 02:51:42.275750 13764 tablet_bootstrap.cc:492] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad: Bootstrap starting.
I20260925 02:51:42.279712 13764 tablet_bootstrap.cc:654] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:42.281153 13764 log.cc:824] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:42.281038 13761 raft_consensus.cc:348] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501a953016ff4745ab448f42616afa67" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 33101 } } peers { permanent_uuid: "60ac9c66c54947f99ad745b97743cd0f" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33331 } } peers { permanent_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44497 } }
I20260925 02:51:42.281493 13761 raft_consensus.cc:374] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:42.281623 13761 raft_consensus.cc:729] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ac9c66c54947f99ad745b97743cd0f, State: Initialized, Role: FOLLOWER
I20260925 02:51:42.282567 13761 consensus_queue.cc:260] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501a953016ff4745ab448f42616afa67" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 33101 } } peers { permanent_uuid: "60ac9c66c54947f99ad745b97743cd0f" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33331 } } peers { permanent_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44497 } }
I20260925 02:51:42.283839 13606 heartbeater.cc:503] Master 127.2.172.126:41507 was elected leader, sending a full tablet report...
I20260925 02:51:42.284384 13761 ts_tablet_manager.cc:1428] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260925 02:51:42.284914 13764 tablet_bootstrap.cc:492] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad: No bootstrap required, opened a new log
I20260925 02:51:42.285357 13764 ts_tablet_manager.cc:1397] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20260925 02:51:42.285801 13762 raft_consensus.cc:348] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501a953016ff4745ab448f42616afa67" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 33101 } } peers { permanent_uuid: "60ac9c66c54947f99ad745b97743cd0f" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33331 } } peers { permanent_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44497 } }
I20260925 02:51:42.286370 13762 raft_consensus.cc:374] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:42.286481 13762 raft_consensus.cc:729] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501a953016ff4745ab448f42616afa67, State: Initialized, Role: FOLLOWER
I20260925 02:51:42.287141 13762 consensus_queue.cc:260] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501a953016ff4745ab448f42616afa67" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 33101 } } peers { permanent_uuid: "60ac9c66c54947f99ad745b97743cd0f" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33331 } } peers { permanent_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44497 } }
I20260925 02:51:42.288961 13762 ts_tablet_manager.cc:1428] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 02:51:42.296399 13764 raft_consensus.cc:348] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501a953016ff4745ab448f42616afa67" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 33101 } } peers { permanent_uuid: "60ac9c66c54947f99ad745b97743cd0f" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33331 } } peers { permanent_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44497 } }
I20260925 02:51:42.296864 13764 raft_consensus.cc:374] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:42.296991 13764 raft_consensus.cc:729] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c247eb4ac0d14b1ba815777fbcbcc4ad, State: Initialized, Role: FOLLOWER
I20260925 02:51:42.297770 13764 consensus_queue.cc:260] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501a953016ff4745ab448f42616afa67" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 33101 } } peers { permanent_uuid: "60ac9c66c54947f99ad745b97743cd0f" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33331 } } peers { permanent_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44497 } }
I20260925 02:51:42.298943 13742 heartbeater.cc:503] Master 127.2.172.126:41507 was elected leader, sending a full tablet report...
I20260925 02:51:42.299762 13764 ts_tablet_manager.cc:1428] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 02:51:42.307454 13768 raft_consensus.cc:482] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:42.307860 13768 raft_consensus.cc:504] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501a953016ff4745ab448f42616afa67" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 33101 } } peers { permanent_uuid: "60ac9c66c54947f99ad745b97743cd0f" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33331 } } peers { permanent_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44497 } }
I20260925 02:51:42.309600 13768 leader_election.cc:288] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ac9c66c54947f99ad745b97743cd0f (127.2.172.66:33331), c247eb4ac0d14b1ba815777fbcbcc4ad (127.2.172.67:44497)
I20260925 02:51:42.317523 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.310188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:54309 (local address 127.2.172.67:44497)
0925 02:51:42.310417 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:42.310425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:42.311313 (+ 888us) server_negotiation.cc:407] Connection header received
0925 02:51:42.311447 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:42.311456 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:42.311530 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:42.311666 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:42.312331 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.313313 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.315416 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.316409 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.316546 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.316605 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:42.317069 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 02:51:42.317255 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:51:42.317761 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.310013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:44497 (local address 127.2.172.65:54309)
0925 02:51:42.311208 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:42.311222 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:42.311331 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:42.311753 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:42.311756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:42.311778 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:42.312153 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:42.312162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.313471 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.313501 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:42.315241 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:42.315255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.316656 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.316663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:42.316858 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.316905 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:51:42.317015 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:51:42.317167 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":1,"thread_start_us":615,"threads_started":1}
I20260925 02:51:42.317837 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.310056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:38583 (local address 127.2.172.66:33331)
0925 02:51:42.310333 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:42.310344 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:42.311201 (+ 857us) server_negotiation.cc:407] Connection header received
0925 02:51:42.311836 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:42.311840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:42.311899 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:42.312009 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:42.312734 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.313701 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.315741 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.316776 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.316958 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.317058 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:42.317416 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 02:51:42.317557 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 02:51:42.317904 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.310002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:33331 (local address 127.2.172.65:38583)
0925 02:51:42.311078 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:42.311097 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:42.311308 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:42.312030 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:42.312035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:42.312061 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:42.312547 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:42.312562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.313870 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.313874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:42.315583 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:42.315598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.317060 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.317067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:42.317236 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.317268 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 02:51:42.317379 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 02:51:42.317501 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":1,"thread_start_us":841,"threads_started":1}
I20260925 02:51:42.319144 13560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d23aaff5df4f139abd81ea645fde7d" candidate_uuid: "501a953016ff4745ab448f42616afa67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ac9c66c54947f99ad745b97743cd0f" is_pre_election: true
I20260925 02:51:42.319065 13696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d23aaff5df4f139abd81ea645fde7d" candidate_uuid: "501a953016ff4745ab448f42616afa67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad" is_pre_election: true
I20260925 02:51:42.319882 13560 raft_consensus.cc:2454] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 501a953016ff4745ab448f42616afa67 in term 0.
I20260925 02:51:42.320055 13696 raft_consensus.cc:2454] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 501a953016ff4745ab448f42616afa67 in term 0.
I20260925 02:51:42.320799 13357 leader_election.cc:302] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 501a953016ff4745ab448f42616afa67, 60ac9c66c54947f99ad745b97743cd0f, c247eb4ac0d14b1ba815777fbcbcc4ad; no voters:
I20260925 02:51:42.321309 13768 raft_consensus.cc:2781] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:42.321446 13768 raft_consensus.cc:482] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:42.321609 13768 raft_consensus.cc:3037] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:42.324064 13768 raft_consensus.cc:504] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501a953016ff4745ab448f42616afa67" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 33101 } } peers { permanent_uuid: "60ac9c66c54947f99ad745b97743cd0f" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33331 } } peers { permanent_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44497 } }
I20260925 02:51:42.324685 13768 leader_election.cc:288] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [CANDIDATE]: Term 1 election: Requested vote from peers 60ac9c66c54947f99ad745b97743cd0f (127.2.172.66:33331), c247eb4ac0d14b1ba815777fbcbcc4ad (127.2.172.67:44497)
I20260925 02:51:42.325294 13696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d23aaff5df4f139abd81ea645fde7d" candidate_uuid: "501a953016ff4745ab448f42616afa67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad"
I20260925 02:51:42.325294 13560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d23aaff5df4f139abd81ea645fde7d" candidate_uuid: "501a953016ff4745ab448f42616afa67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ac9c66c54947f99ad745b97743cd0f"
I20260925 02:51:42.325526 13560 raft_consensus.cc:3037] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:42.325515 13696 raft_consensus.cc:3037] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:42.328853 13560 raft_consensus.cc:2454] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501a953016ff4745ab448f42616afa67 in term 1.
I20260925 02:51:42.328853 13696 raft_consensus.cc:2454] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501a953016ff4745ab448f42616afa67 in term 1.
I20260925 02:51:42.329384 13359 leader_election.cc:302] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 501a953016ff4745ab448f42616afa67, 60ac9c66c54947f99ad745b97743cd0f; no voters:
I20260925 02:51:42.329767 13768 raft_consensus.cc:2781] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:42.330209 13768 raft_consensus.cc:686] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 1 LEADER]: Becoming Leader. State: Replica: 501a953016ff4745ab448f42616afa67, State: Running, Role: LEADER
I20260925 02:51:42.330696 13768 consensus_queue.cc:237] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501a953016ff4745ab448f42616afa67" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 33101 } } peers { permanent_uuid: "60ac9c66c54947f99ad745b97743cd0f" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33331 } } peers { permanent_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44497 } }
I20260925 02:51:42.335240 13278 catalog_manager.cc:5495] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 501a953016ff4745ab448f42616afa67 (127.2.172.65). New cstate: current_term: 1 leader_uuid: "501a953016ff4745ab448f42616afa67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501a953016ff4745ab448f42616afa67" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 33101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60ac9c66c54947f99ad745b97743cd0f" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44497 } health_report { overall_health: UNKNOWN } } }
W20260925 02:51:42.342746 13607 tablet.cc:2369] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:51:42.352916 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.347467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.2.172.65:33101)
0925 02:51:42.347732 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:42.347739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:42.347759 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:51:42.348003 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:42.348007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:42.348085 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:42.348241 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:42.348615 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.349350 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.350673 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.351069 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.351201 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.351735 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:42.352351 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:42.352526 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:42.352577 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 02:51:42.352659 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260925 02:51:42.361486 13696 raft_consensus.cc:1261] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad [term 1 FOLLOWER]: Refusing update from remote peer 501a953016ff4745ab448f42616afa67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:42.361511 13560 raft_consensus.cc:1261] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f [term 1 FOLLOWER]: Refusing update from remote peer 501a953016ff4745ab448f42616afa67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:42.362491 13772 consensus_queue.cc:1035] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c247eb4ac0d14b1ba815777fbcbcc4ad" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:42.362849 13768 consensus_queue.cc:1035] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ac9c66c54947f99ad745b97743cd0f" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:42.378638 13777 mvcc.cc:205] Tried to move back new op lower bound from 7333088060861628416 to 7333088060750286848. Current Snapshot: MvccSnapshot[applied={T|T < 7333088060861628416 or (T in {7333088060861628416})}]
I20260925 02:51:42.390110 13280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:42.390497 13280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:42.394357 13280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c9703a490d54cc2b69667049b580a71: Sending DeleteTablet for 3 replicas of tablet 05d23aaff5df4f139abd81ea645fde7d
I20260925 02:51:42.395905 13404 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d23aaff5df4f139abd81ea645fde7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:42 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:41854
I20260925 02:51:42.395900 13676 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d23aaff5df4f139abd81ea645fde7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:42 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:54590
I20260925 02:51:42.396165 13539 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d23aaff5df4f139abd81ea645fde7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:42 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:43184
I20260925 02:51:42.399022 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 13341
I20260925 02:51:42.409536 13784 tablet_replica.cc:330] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67: stopping tablet replica
I20260925 02:51:42.410182 13784 raft_consensus.cc:2229] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:51:42.410929 13784 raft_consensus.cc:2258] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:42.422366 13784 ts_tablet_manager.cc:1905] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:42.427167 13786 tablet_replica.cc:330] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f: stopping tablet replica
Sep 25 02:51:42 dist-test-slave-g1xh krb5kdc[13234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:51:42.432221 13786 raft_consensus.cc:2229] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:42.432938 13786 raft_consensus.cc:2258] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:42.434523 13784 ts_tablet_manager.cc:1918] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:42.437917 13785 tablet_replica.cc:330] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad: stopping tablet replica
I20260925 02:51:42.437964 13786 ts_tablet_manager.cc:1905] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:42.438475 13785 raft_consensus.cc:2229] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:42.438762 13784 log.cc:1196] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/05d23aaff5df4f139abd81ea645fde7d
I20260925 02:51:42.439061 13785 raft_consensus.cc:2258] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:42.439785 13784 ts_tablet_manager.cc:1939] T 05d23aaff5df4f139abd81ea645fde7d P 501a953016ff4745ab448f42616afa67: Deleting consensus metadata
I20260925 02:51:42.444986 13785 ts_tablet_manager.cc:1905] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:42.446484 13786 ts_tablet_manager.cc:1918] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:42.446663 13786 log.cc:1196] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/05d23aaff5df4f139abd81ea645fde7d
I20260925 02:51:42.447252 13786 ts_tablet_manager.cc:1939] T 05d23aaff5df4f139abd81ea645fde7d P 60ac9c66c54947f99ad745b97743cd0f: Deleting consensus metadata
I20260925 02:51:42.444092 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.401498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.2.172.65:33101)
0925 02:51:42.401757 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:42.401766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:42.411889 (+ 10123us) server_negotiation.cc:407] Connection header received
0925 02:51:42.412138 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:42.412147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:42.412283 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:42.412416 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:42.416756 (+ 4340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.418416 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.423721 (+ 5305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.424878 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.425022 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.434752 (+ 9730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:42.434775 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:42.434805 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:42.434870 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:42.439928 (+ 5058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:42.440878 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:42.440889 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:42.440893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:42.441062 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:42.441530 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:42.441535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:42.441539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:42.441982 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:42.442252 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:42.443438 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0925 02:51:42.443707 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:51:42.448371 13465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:42.451416 13785 ts_tablet_manager.cc:1918] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:42.451671 13785 log.cc:1196] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/05d23aaff5df4f139abd81ea645fde7d
I20260925 02:51:42.452195 13785 ts_tablet_manager.cc:1939] T 05d23aaff5df4f139abd81ea645fde7d P c247eb4ac0d14b1ba815777fbcbcc4ad: Deleting consensus metadata
I20260925 02:51:42.452924 13265 catalog_manager.cc:4841] TS 60ac9c66c54947f99ad745b97743cd0f (127.2.172.66:33331): tablet 05d23aaff5df4f139abd81ea645fde7d (table test-table [id=fcda88c26ba64ed38f4a573aee00029f]) successfully deleted
I20260925 02:51:42.454555 13263 catalog_manager.cc:4841] TS c247eb4ac0d14b1ba815777fbcbcc4ad (127.2.172.67:44497): tablet 05d23aaff5df4f139abd81ea645fde7d (table test-table [id=fcda88c26ba64ed38f4a573aee00029f]) successfully deleted
I20260925 02:51:42.553007 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 13341
I20260925 02:51:42.553257 13265 catalog_manager.cc:4841] TS 501a953016ff4745ab448f42616afa67 (127.2.172.65:33101): tablet 05d23aaff5df4f139abd81ea645fde7d (table test-table [id=fcda88c26ba64ed38f4a573aee00029f]) successfully deleted
I20260925 02:51:42.573467 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 13476
Sep 25 02:51:42 dist-test-slave-g1xh krb5kdc[13234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:51:42.591857 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.574314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.2.172.66:33331)
0925 02:51:42.574518 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:42.574526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:42.574630 (+ 104us) server_negotiation.cc:407] Connection header received
0925 02:51:42.574882 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:42.574887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:42.574952 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:42.575043 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:42.576674 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.577408 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.578699 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.579144 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.579307 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.584159 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:42.584178 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:42.584196 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:42.584241 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:42.588304 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:42.589225 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:42.589233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:42.589236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:42.589386 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:42.589802 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:42.589807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:42.589812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:42.590281 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:42.590605 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:42.590916 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 02:51:42.591352 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:51:42.592734 13601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:42.683979 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 13476
I20260925 02:51:42.703428 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 13612
Sep 25 02:51:42 dist-test-slave-g1xh krb5kdc[13234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:51:42.718724 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.704361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.2.172.67:44497)
0925 02:51:42.704549 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:42.704558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:42.704581 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:51:42.704818 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:42.704823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:42.704891 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:42.705000 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:42.706285 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.706994 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.708367 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:42.708772 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:42.708903 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:42.713135 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:42.713149 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:42.713166 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:42.713204 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:42.715997 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:42.716594 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:42.716601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:42.716603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:42.716746 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:42.717092 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:42.717097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:42.717099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:42.717563 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:42.717812 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:42.718155 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 02:51:42.718436 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:51:42.719379 13737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:42.811682 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 13612
I20260925 02:51:42.829620 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 13247
I20260925 02:51:42.830829 13310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:51:42.940485 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 13247
2026-09-25T02:51:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4366 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:51:43 dist-test-slave-g1xh krb5kdc[13794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:51:43 dist-test-slave-g1xh krb5kdc[13794](info): set up 2 sockets
Sep 25 02:51:43 dist-test-slave-g1xh krb5kdc[13794](info): commencing operation
krb5kdc: starting...
W20260925 02:51:44.078742 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:51:44 dist-test-slave-g1xh krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:51:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:51:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/rangeradmin_127.2.172.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/rangeradmin_127.2.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/rangerlookup_127.2.172.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/rangerlookup_127.2.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/HTTP_127.2.172.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/HTTP_127.2.172.84@KRBTEST.COM.keytab.
I20260925 02:51:44.464296 2737 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task450sZB/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/postgres -l logfile start
2026-09-25 02:51:47.385 UTC [13824] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:51:47.385 UTC [13824] LOG: listening on IPv4 address "127.2.172.84", port 45009
2026-09-25 02:51:47.389 UTC [13824] LOG: listening on Unix socket "/tmp/.s.PGSQL.45009"
2026-09-25 02:51:47.416 UTC [13826] LOG: database system was shut down at 2026-09-25 02:51:45 UTC
2026-09-25 02:51:47.420 UTC [13824] LOG: database system is ready to accept connections
I20260925 02:51:48.471899 2737 mini_postgres.cc:96] Postgres bound to 45009
2026-09-25 02:51:48.500 UTC [13835] FATAL: database "slave" does not exist
127.2.172.84:45009 - accepting connections
I20260925 02:51:48.501828 2737 mini_ranger.cc:161] Starting Ranger...
I20260925 02:51:48.538937 2737 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:51:49.090160 2737 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:51:49.090395 2737 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-admin
2026-09-25 02:51:49,639 [I] DB FLAVOR :POSTGRES
2026-09-25 02:51:49,640 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:51:49,640 [I] Checking connection..
2026-09-25 02:51:49,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:51:49,924 [I] Checking connection passed.
2026-09-25 02:51:49,924 [I] --------- Verifying version history table ---------
2026-09-25 02:51:49,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:51:50,186 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:51:50,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:51:50,448 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:51:50,448 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:51:50,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:51:50.718 UTC [13900] WARNING: there is no transaction in progress
2026-09-25 02:51:50,725 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:51:50,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:51:50,968 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:51:50,968 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:51:50,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:51:51,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:51:51,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g1xh.c.gcp-upstream.internal','N') ;"
2026-09-25 02:51:51,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:51:52,055 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:51:52,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:51:52,318 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:51:52,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:51:52,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:51:52,834 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:51:52,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:51:53.865 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:53.883 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:53.900 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:53.917 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:53.935 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:53.957 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:54.015 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:54.023 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:54.039 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:54.050 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:54.061 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:54.077 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:54.088 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:54.095 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:54.102 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:54.111 UTC [14008] WARNING: there is no transaction in progress
2026-09-25 02:51:54,854 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:51:54,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:51:55,094 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:51:55,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:51:55,364 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:51:55,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:51:55,637 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:51:55,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:51:55,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:51:56,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g1xh.c.gcp-upstream.internal';"
2026-09-25 02:51:56,399 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:51:56,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:51:56,655 [I] DB_PATCHES have already been applied
I20260925 02:51:56.661815 2737 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-admin:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task450sZB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:51:56.662045 2737 mini_ranger.cc:193] Using host: 127.2.172.84
I20260925 02:51:56.680172 2737 mini_ranger.cc:227] Ranger admin URL: http://127.2.172.84:44369
Sep 25, 2026 2:51:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:51:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:51:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:51:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:51:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:51:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:51:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:51:57.821587 2737 mini_ranger.cc:160] Time spent starting Ranger: real 9.320s user 0.000s sys 0.059s
Sep 25, 2026 2:52:20 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:52:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 2:52:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:52:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:52:20 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:52:21 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:52:21.877872 2737 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/rangerkms_127.2.172.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/rangerkms_127.2.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.172.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.172.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.172.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/HTTP_127.2.172.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.172.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/HTTP_127.2.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:52:22 dist-test-slave-g1xh krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304742, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:52:22.084053 2737 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:52:22.110782 2737 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:52:23.061532 2737 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:52:23.061647 2737 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-kms
I20260925 02:52:23.061815 2737 mini_ranger_kms.cc:227] Using postgres at 127.2.172.84:45009
2026-09-25 02:52:24,049 [I] DB FLAVOR :POSTGRES
2026-09-25 02:52:24,049 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:52:24,049 [I] Checking connection
2026-09-25 02:52:24,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:52:24,315 [I] connection success
2026-09-25 02:52:24,316 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:52:24,316 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:52:24,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:52:24,558 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:52:24,558 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:52:24,558 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:52:24,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:45009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:52:24,822 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:52:25.315375 2737 mini_ranger_kms.cc:310] Created kms service
I20260925 02:52:25.526368 2737 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:52:25.576270 2737 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:52:25 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304742, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:52:26.026268 2737 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:52:26.026648 2737 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-kms:/tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:52:26.026715 2737 mini_ranger_kms.cc:240] Using host: 127.2.172.84
I20260925 02:52:26.047945 2737 mini_ranger_kms.cc:276] Ranger KMS PID: 14191
I20260925 02:52:26.048157 2737 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.172.84:40515
Sep 25, 2026 2:52:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:52:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:52:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:52:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:52:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:52:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 02:52:28 dist-test-slave-g1xh krb5kdc[13794](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304748, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.172.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:28.416802 2737 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.333s user 0.174s sys 0.509s
I20260925 02:52:28.417083 2737 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:52:28.417320 2737 mini_ranger_kms.cc:401] 127.2.172.84:40515/kms/v1/keys
Sep 25 02:52:28 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304748, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.172.84@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
Sep 25 02:52:29 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304748, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.172.84@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
Sep 25 02:52:31 dist-test-slave-g1xh krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:52:31.520308 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:43509
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:43509
--ranger_config_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.172.84:40515/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:31.885756 14241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:31.886075 14241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:31.886235 14241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:31.893750 14241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:31.893873 14241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:31.893959 14241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:31.894055 14241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:31.894138 14241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:31.904989 14241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.172.84:40515/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:43509
--ranger_config_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:43509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:52:31.906368 14241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:31.907743 14241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:31.921499 14246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:31.921532 14247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:31.922380 14249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:31.922955 14241 server_base.cc:1034] running on GCE node
I20260925 02:52:31.923892 14241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:31.925436 14241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:31.926671 14241 hybrid_clock.cc:648] HybridClock initialized: now 1790304751926624 us; error 36 us; skew 500 ppm
I20260925 02:52:31.927088 14241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:52:31 dist-test-slave-g1xh krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304751, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:31.936159 14241 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:52:31.937956 14241 webserver.cc:460] Webserver started at http://127.2.172.126:39355/ using document root <none> and password file <none>
I20260925 02:52:31.938652 14241 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:31.938788 14241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:31.939141 14241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:52:31 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304751, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:52:32.000694 14241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "0d213da55e0747b39be74207b2decf09"
format_stamp: "Formatted at 2026-09-25 02:52:31 on dist-test-slave-g1xh"
server_key: "15894d5f740b3a8f13342e0dc50239ad"
server_key_iv: "84dba8e3123ec79f57973da3595b562c"
server_key_version: "kuduclusterkey@0"
I20260925 02:52:32.001776 14241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "0d213da55e0747b39be74207b2decf09"
format_stamp: "Formatted at 2026-09-25 02:52:31 on dist-test-slave-g1xh"
server_key: "15894d5f740b3a8f13342e0dc50239ad"
server_key_iv: "84dba8e3123ec79f57973da3595b562c"
server_key_version: "kuduclusterkey@0"
I20260925 02:52:32.008805 14241 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260925 02:52:32.029085 14256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:32.030844 14241 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:52:32.031054 14241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "0d213da55e0747b39be74207b2decf09"
format_stamp: "Formatted at 2026-09-25 02:52:31 on dist-test-slave-g1xh"
server_key: "15894d5f740b3a8f13342e0dc50239ad"
server_key_iv: "84dba8e3123ec79f57973da3595b562c"
server_key_version: "kuduclusterkey@0"
I20260925 02:52:32.031347 14241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:32.054724 14241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:32.058509 14241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:32.058763 14241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:32.078424 14241 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:43509
I20260925 02:52:32.078447 14307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:43509 every 8 connection(s)
I20260925 02:52:32.079882 14241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:32.081453 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 14241
I20260925 02:52:32.081660 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:52:32.082316 2737 external_mini_cluster.cc:1428] Setting key 3fa367755e2110a5391e0427ef281387
I20260925 02:52:32.084386 14309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:52:32 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:52:32.099145 14309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09: Bootstrap starting.
I20260925 02:52:32.103222 14309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:32.104389 14309 log.cc:824] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:32.107401 14309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09: No bootstrap required, opened a new log
I20260925 02:52:32.107543 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:32.084428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.2.172.126:43509)
0925 02:52:32.085830 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:32.085842 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:32.085883 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:52:32.087480 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:32.087504 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:32.088008 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:32.088360 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:32.090075 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:32.091899 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:32.093671 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:32.094498 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:32.094642 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:32.099426 (+ 4784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:32.099436 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:32.099453 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:32.099493 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:32.103228 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:32.104025 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:32.104034 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:32.104043 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:32.104218 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:32.104735 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:32.104740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:32.104744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:32.105295 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:32.105632 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:32.106086 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 02:52:32.106504 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":939,"threads_started":1}
I20260925 02:52:32.116595 14309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d213da55e0747b39be74207b2decf09" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43509 } }
I20260925 02:52:32.117014 14309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:32.117126 14309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d213da55e0747b39be74207b2decf09, State: Initialized, Role: FOLLOWER
I20260925 02:52:32.117789 14309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d213da55e0747b39be74207b2decf09" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43509 } }
I20260925 02:52:32.118048 14309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:32.118201 14309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:32.118407 14309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:32.120836 14309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d213da55e0747b39be74207b2decf09" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43509 } }
I20260925 02:52:32.121457 14309 leader_election.cc:302] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d213da55e0747b39be74207b2decf09; no voters:
I20260925 02:52:32.122121 14309 leader_election.cc:288] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:32.122282 14314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:32.123450 14314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [term 1 LEADER]: Becoming Leader. State: Replica: 0d213da55e0747b39be74207b2decf09, State: Running, Role: LEADER
I20260925 02:52:32.124017 14314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d213da55e0747b39be74207b2decf09" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43509 } }
I20260925 02:52:32.124434 14309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:32.127200 14316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d213da55e0747b39be74207b2decf09. Latest consensus state: current_term: 1 leader_uuid: "0d213da55e0747b39be74207b2decf09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d213da55e0747b39be74207b2decf09" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43509 } } }
I20260925 02:52:32.127812 14316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:32.127291 14315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d213da55e0747b39be74207b2decf09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d213da55e0747b39be74207b2decf09" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43509 } } }
I20260925 02:52:32.127950 14315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:32.135962 14309 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:52:32.136180 14320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:32.149858 14320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:32.162515 14320 catalog_manager.cc:1347] Generated new cluster ID: e64a9855d7ef42cdaaf9ff82694dc07c
I20260925 02:52:32.162688 14320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:32.185985 14320 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:32.188138 14320 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:32.197804 14320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09: Generated new TSK 0
I20260925 02:52:32.198932 14320 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:52:32.941596 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.172.84:40515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43509
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:52:33 dist-test-slave-g1xh krb5kdc[13794](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 02:52:33.368634 14355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:33.369084 14355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:33.369324 14355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:33.381309 14355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:33.381477 14355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:33.381759 14355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:52:33.396366 14355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.172.84:40515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:52:33.397761 14355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:33.399189 14355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:33.416779 14365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:33.418016 14366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:33.429036 14368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:33.429555 14355 server_base.cc:1034] running on GCE node
I20260925 02:52:33.430245 14355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:33.431372 14355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:33.432641 14355 hybrid_clock.cc:648] HybridClock initialized: now 1790304753432565 us; error 57 us; skew 500 ppm
I20260925 02:52:33.433060 14355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:52:33 dist-test-slave-g1xh krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:33.437845 14355 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:52:33.444983 14355 webserver.cc:460] Webserver started at http://127.2.172.65:45335/ using document root <none> and password file <none>
I20260925 02:52:33.446026 14355 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:33.446173 14355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:33.446557 14355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:52:33 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:52:33.465607 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "e454dc0e1e4d4806972f696ab589b8e2"
format_stamp: "Formatted at 2026-09-25 02:52:33 on dist-test-slave-g1xh"
server_key: "7d0c35eba986275e0773940a90a3828a"
server_key_iv: "cf94e5aa55f91c50f24887371b234c5d"
server_key_version: "kuduclusterkey@0"
I20260925 02:52:33.467101 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "e454dc0e1e4d4806972f696ab589b8e2"
format_stamp: "Formatted at 2026-09-25 02:52:33 on dist-test-slave-g1xh"
server_key: "7d0c35eba986275e0773940a90a3828a"
server_key_iv: "cf94e5aa55f91c50f24887371b234c5d"
server_key_version: "kuduclusterkey@0"
I20260925 02:52:33.492125 14355 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.002s sys 0.018s
I20260925 02:52:33.528286 14375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:33.530508 14355 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.009s sys 0.008s
I20260925 02:52:33.530799 14355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "e454dc0e1e4d4806972f696ab589b8e2"
format_stamp: "Formatted at 2026-09-25 02:52:33 on dist-test-slave-g1xh"
server_key: "7d0c35eba986275e0773940a90a3828a"
server_key_iv: "cf94e5aa55f91c50f24887371b234c5d"
server_key_version: "kuduclusterkey@0"
I20260925 02:52:33.531134 14355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:33.586067 14355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:33.591933 14355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:33.592271 14355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:33.593334 14355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:33.594847 14355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:33.594920 14355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:33.595048 14355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:33.595108 14355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:33.680680 14355 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:39279
I20260925 02:52:33.680728 14489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:39279 every 8 connection(s)
I20260925 02:52:33.682541 14355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:52:33.686084 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 14355
I20260925 02:52:33.686344 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:52:33.687322 2737 external_mini_cluster.cc:1428] Setting key 57261fc183ac0d742d59be20ba89a8a0
Sep 25 02:52:33 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
I20260925 02:52:33.740900 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.695466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:44599 (local address 127.2.172.126:43509)
0925 02:52:33.708868 (+ 13402us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.708875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.708897 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:52:33.708963 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.708968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.709070 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.709197 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.711359 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.712561 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.718708 (+ 6147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.719416 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.719580 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.729434 (+ 9854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.729443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.729449 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.729508 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.734844 (+ 5336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.736221 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.736227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.736231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.736402 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.737030 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.737036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.737039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.737611 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.737756 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.740255 (+ 2499us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.740493 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13254,"thread_start_us":13141,"threads_started":1}
I20260925 02:52:33.742921 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.694776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43509 (local address 127.2.172.65:44599)
0925 02:52:33.704262 (+ 9486us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.704332 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:33.708078 (+ 3746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:33.709534 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:33.709548 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:33.710187 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:33.711124 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.711151 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.712749 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.712754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.714835 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.714861 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.719699 (+ 4838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.719724 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.719967 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.724337 (+ 4370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:33.724380 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:33.729088 (+ 4708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:33.735101 (+ 6013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.735112 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.735135 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.735909 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.736548 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.736553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.736556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.736872 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.737890 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:33.737899 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:33.738982 (+ 1083us) client_negotiation.cc:769] Sending connection context
0925 02:52:33.739311 (+ 329us) client_negotiation.cc:240] Negotiation successful
0925 02:52:33.739979 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9130,"thread_start_us":8892,"threads_started":1}
I20260925 02:52:33.745342 14490 heartbeater.cc:347] Connected to a master server at 127.2.172.126:43509
I20260925 02:52:33.745705 14490 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:33.746630 14490 heartbeater.cc:511] Master 127.2.172.126:43509 requested a full tablet report, sending...
I20260925 02:52:33.749327 14273 ts_manager.cc:194] Registered new tserver with Master: e454dc0e1e4d4806972f696ab589b8e2 (127.2.172.65:39279)
I20260925 02:52:33.751406 14273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:44599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:52:33.780323 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.172.84:40515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43509
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:52:33 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
Sep 25 02:52:34 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
W20260925 02:52:34.313199 14496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:34.313752 14496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:34.313977 14496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:34.326483 14496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:34.326720 14496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:34.326960 14496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:52:34.342989 14496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.172.84:40515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:52:34.345077 14496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:34.346823 14496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:34.377892 14503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:34.378873 14504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:34.382766 14506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:34.383425 14496 server_base.cc:1034] running on GCE node
I20260925 02:52:34.384296 14496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:34.394874 14496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:34.396754 14496 hybrid_clock.cc:648] HybridClock initialized: now 1790304754396638 us; error 127 us; skew 500 ppm
I20260925 02:52:34.397330 14496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:52:34 dist-test-slave-g1xh krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304754, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:34.402138 14496 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:52:34.411840 14496 webserver.cc:460] Webserver started at http://127.2.172.66:35319/ using document root <none> and password file <none>
I20260925 02:52:34.413053 14496 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:34.413290 14496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:34.413890 14496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:52:34 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304754, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:52:34.436206 14496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "f44be1dd515b4815bf0b8d8a2c87522d"
format_stamp: "Formatted at 2026-09-25 02:52:34 on dist-test-slave-g1xh"
server_key: "0c300b4084a023b166dbbce7fbed082c"
server_key_iv: "7cc56f7fcbecdbad280c46d5c7eb45d9"
server_key_version: "kuduclusterkey@0"
I20260925 02:52:34.437350 14496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "f44be1dd515b4815bf0b8d8a2c87522d"
format_stamp: "Formatted at 2026-09-25 02:52:34 on dist-test-slave-g1xh"
server_key: "0c300b4084a023b166dbbce7fbed082c"
server_key_iv: "7cc56f7fcbecdbad280c46d5c7eb45d9"
server_key_version: "kuduclusterkey@0"
I20260925 02:52:34.444548 14496 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260925 02:52:34.474577 14514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:34.479916 14496 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.001s
I20260925 02:52:34.480268 14496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "f44be1dd515b4815bf0b8d8a2c87522d"
format_stamp: "Formatted at 2026-09-25 02:52:34 on dist-test-slave-g1xh"
server_key: "0c300b4084a023b166dbbce7fbed082c"
server_key_iv: "7cc56f7fcbecdbad280c46d5c7eb45d9"
server_key_version: "kuduclusterkey@0"
I20260925 02:52:34.480786 14496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:34.506506 14496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:34.512144 14496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:34.512507 14496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:34.513916 14496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:34.515595 14496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:34.515720 14496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 02:52:34.515889 14496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:34.515965 14496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:34.560053 14496 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:35209
I20260925 02:52:34.560073 14634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:35209 every 8 connection(s)
I20260925 02:52:34.561410 14496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:52:34.569595 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 14496
I20260925 02:52:34.569774 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:52:34.570355 2737 external_mini_cluster.cc:1428] Setting key 261a216aae8a099b4cf196cdd1c72206
Sep 25 02:52:34 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304754, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:52:34.583256 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:34.564224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:34109 (local address 127.2.172.126:43509)
0925 02:52:34.564688 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:34.564693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:34.565892 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 02:52:34.567924 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:34.567930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:34.568039 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:34.568163 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:34.569672 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.570544 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.572903 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.573327 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.573471 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:34.577104 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:34.577110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:34.577113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:34.577147 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:34.579964 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:34.580755 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:34.580766 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:34.580769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:34.580895 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:34.581384 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:34.581389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:34.581391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:34.581680 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:34.581849 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:34.582608 (+ 759us) server_negotiation.cc:299] Negotiation successful
0925 02:52:34.582862 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":250,"threads_started":1}
I20260925 02:52:34.584614 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:34.564835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43509 (local address 127.2.172.66:34109)
0925 02:52:34.565691 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:34.565739 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:34.567513 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:34.568418 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:34.568427 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:34.568892 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:34.569492 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:34.569510 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.570710 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.570715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:34.572744 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:34.572757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.573594 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.573604 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:34.573848 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:34.574599 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:34.574623 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:34.576871 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:34.580125 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:34.580135 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:34.580155 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:34.580611 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:34.581013 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:34.581018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:34.581021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:34.581245 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:34.581831 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:34.581839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:34.582211 (+ 372us) client_negotiation.cc:769] Sending connection context
0925 02:52:34.582495 (+ 284us) client_negotiation.cc:240] Negotiation successful
0925 02:52:34.583050 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":333,"threads_started":1}
I20260925 02:52:34.586215 14635 heartbeater.cc:347] Connected to a master server at 127.2.172.126:43509
I20260925 02:52:34.586613 14635 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:34.587458 14635 heartbeater.cc:511] Master 127.2.172.126:43509 requested a full tablet report, sending...
I20260925 02:52:34.589172 14273 ts_manager.cc:194] Registered new tserver with Master: f44be1dd515b4815bf0b8d8a2c87522d (127.2.172.66:35209)
I20260925 02:52:34.590011 14273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:34109
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:52:34.628783 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.172.84:40515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43509
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:52:34.755302 14490 heartbeater.cc:503] Master 127.2.172.126:43509 was elected leader, sending a full tablet report...
W20260925 02:52:34.972388 14641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:34.972666 14641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:34.972846 14641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:34.980476 14641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:34.980602 14641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:34.980819 14641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:52:34.991014 14641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.172.84:40515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:52:34.992462 14641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:34.993815 14641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:35.007967 14649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:35.008005 14647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:35.008579 14646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:35.013185 14641 server_base.cc:1034] running on GCE node
I20260925 02:52:35.013806 14641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:35.014755 14641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:35.015969 14641 hybrid_clock.cc:648] HybridClock initialized: now 1790304755015919 us; error 33 us; skew 500 ppm
I20260925 02:52:35.016263 14641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:52:35 dist-test-slave-g1xh krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304755, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:35.019510 14641 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:52:35.021302 14641 webserver.cc:460] Webserver started at http://127.2.172.67:34233/ using document root <none> and password file <none>
I20260925 02:52:35.022119 14641 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:35.022258 14641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:35.022593 14641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:52:35 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304755, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:52:35.041777 14641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "d1caf8ad5eef4dd4acb1fcd953203684"
format_stamp: "Formatted at 2026-09-25 02:52:35 on dist-test-slave-g1xh"
server_key: "393b41bb5c3b2726702ac81af1e96ee5"
server_key_iv: "71e8ba2250e33417474ed15d7986058b"
server_key_version: "kuduclusterkey@0"
I20260925 02:52:35.043166 14641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "d1caf8ad5eef4dd4acb1fcd953203684"
format_stamp: "Formatted at 2026-09-25 02:52:35 on dist-test-slave-g1xh"
server_key: "393b41bb5c3b2726702ac81af1e96ee5"
server_key_iv: "71e8ba2250e33417474ed15d7986058b"
server_key_version: "kuduclusterkey@0"
I20260925 02:52:35.049780 14641 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260925 02:52:35.066435 14656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:35.068367 14641 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:52:35.068573 14641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "d1caf8ad5eef4dd4acb1fcd953203684"
format_stamp: "Formatted at 2026-09-25 02:52:35 on dist-test-slave-g1xh"
server_key: "393b41bb5c3b2726702ac81af1e96ee5"
server_key_iv: "71e8ba2250e33417474ed15d7986058b"
server_key_version: "kuduclusterkey@0"
I20260925 02:52:35.068835 14641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:35.093227 14641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:35.097316 14641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:35.097573 14641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:35.098635 14641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:35.100052 14641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:35.100131 14641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:35.100212 14641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:35.100262 14641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:35.135535 14641 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:35653
I20260925 02:52:35.135773 14770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:35653 every 8 connection(s)
I20260925 02:52:35.137018 14641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:52:35.144467 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 14641
I20260925 02:52:35.144686 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:52:35.145298 2737 external_mini_cluster.cc:1428] Setting key 13116b9176110d0c5a00e230dbc344cf
Sep 25 02:52:35 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304755, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:52:35.158215 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.139754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:33453 (local address 127.2.172.126:43509)
0925 02:52:35.140292 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:35.140297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:35.141091 (+ 794us) server_negotiation.cc:407] Connection header received
0925 02:52:35.143013 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:35.143020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:35.143126 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:35.143257 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:35.145074 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.145906 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.147909 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.148474 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.148608 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.152118 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:35.152124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:35.152128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:35.152179 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:35.154980 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:35.155929 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:35.155941 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:35.155943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:35.156107 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:35.156574 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:35.156578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:35.156580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:35.156926 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:35.157048 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:35.157724 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 02:52:35.157938 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":309,"threads_started":1}
I20260925 02:52:35.159343 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.140205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43509 (local address 127.2.172.67:33453)
0925 02:52:35.140916 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:35.140964 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:35.142725 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:35.143500 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:35.143508 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:35.143979 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:35.144870 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:35.144898 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.146123 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.146127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:35.147725 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:35.147750 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.148717 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.148725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:35.148965 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.149573 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:35.149601 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:35.151858 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:35.155127 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:35.155135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:35.155150 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:35.155783 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:35.156210 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:35.156213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:35.156215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:35.156436 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:35.157053 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:35.157059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:35.157399 (+ 340us) client_negotiation.cc:769] Sending connection context
0925 02:52:35.157632 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 02:52:35.158034 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":277,"threads_started":1}
I20260925 02:52:35.160980 14771 heartbeater.cc:347] Connected to a master server at 127.2.172.126:43509
I20260925 02:52:35.161324 14771 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:35.162034 14771 heartbeater.cc:511] Master 127.2.172.126:43509 requested a full tablet report, sending...
I20260925 02:52:35.163431 14273 ts_manager.cc:194] Registered new tserver with Master: d1caf8ad5eef4dd4acb1fcd953203684 (127.2.172.67:35653)
I20260925 02:52:35.164310 14273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:33453
I20260925 02:52:35.173712 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:52:35.192248 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.178826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.2.172.126:43509)
0925 02:52:35.179069 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:35.179078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:35.179455 (+ 377us) server_negotiation.cc:407] Connection header received
0925 02:52:35.179668 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:35.179673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:35.179776 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:35.179861 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:35.181291 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.182008 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.183378 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.183908 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.184016 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.186575 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:35.186580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:35.186585 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:35.186619 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:35.189413 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:35.190182 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:35.190186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:35.190188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:35.190321 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:35.190728 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:35.190733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:35.190736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:35.191008 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:35.191211 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:35.191686 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 02:52:35.191882 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:52:35.196633 14273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:52:35.203933 14273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:52:35.228588 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.216923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:39279 (local address 127.0.0.1:45836)
0925 02:52:35.218180 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:35.218227 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:35.218451 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:35.219989 (+ 1538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:35.219999 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:35.220032 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:35.220510 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:35.220525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.222504 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.222511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:35.225350 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:35.225363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.227380 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.227387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:35.227676 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.227740 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 02:52:35.227867 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 02:52:35.228082 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":2,"thread_start_us":768,"threads_started":1}
I20260925 02:52:35.229208 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.217193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.2.172.65:39279)
0925 02:52:35.219236 (+ 2043us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:35.219248 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:35.219282 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:52:35.219403 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:35.219414 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:35.219749 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:35.219954 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:35.220735 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.222343 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.225566 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.227144 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.228101 (+ 957us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.228357 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:35.228427 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:52:35.228605 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1818,"thread_start_us":1639,"threads_started":1}
I20260925 02:52:35.231837 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.217070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:35209 (local address 127.0.0.1:45362)
0925 02:52:35.219934 (+ 2864us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:35.219952 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:35.220072 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:35.221083 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:35.221088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:35.221111 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:35.222689 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:35.222707 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.224734 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.224742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:35.229134 (+ 4392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:35.229152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.230967 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.230975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:35.231149 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.231190 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 02:52:35.231306 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 02:52:35.231434 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2726,"mutex_wait_us":832,"thread_start_us":2564,"threads_started":1}
I20260925 02:52:35.231931 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.217407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.2.172.66:35209)
0925 02:52:35.219445 (+ 2038us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:35.219455 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:35.220221 (+ 766us) server_negotiation.cc:407] Connection header received
0925 02:52:35.220333 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:35.220344 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:35.220590 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:35.220775 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:35.222923 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.224528 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.229550 (+ 5022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.230747 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.230864 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.230993 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:35.231365 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 02:52:35.231542 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1891,"thread_start_us":1827,"threads_started":1}
I20260925 02:52:35.233245 14424 tablet_service.cc:1467] Processing CreateTablet for tablet 523c2dcc0aae489a9ecb5c36fcd5880c (DEFAULT_TABLE table=test-table [id=6e0482fcdace4a64a0e988059e6a0a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:35.234959 14424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523c2dcc0aae489a9ecb5c36fcd5880c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:35.235533 14569 tablet_service.cc:1467] Processing CreateTablet for tablet 523c2dcc0aae489a9ecb5c36fcd5880c (DEFAULT_TABLE table=test-table [id=6e0482fcdace4a64a0e988059e6a0a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:35.236940 14569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523c2dcc0aae489a9ecb5c36fcd5880c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:35.237221 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.220682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.2.172.67:35653)
0925 02:52:35.227979 (+ 7297us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:35.227987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:35.228615 (+ 628us) server_negotiation.cc:407] Connection header received
0925 02:52:35.228736 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:35.228747 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:35.229009 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:35.229214 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:35.231027 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.232741 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.234758 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.236042 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.236198 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.236424 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:35.236701 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 02:52:35.236857 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7118,"thread_start_us":6920,"threads_started":1}
I20260925 02:52:35.237578 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.220042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:35653 (local address 127.0.0.1:41874)
0925 02:52:35.228340 (+ 8298us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:35.228360 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:35.228496 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:35.229577 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:35.229582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:35.229605 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:35.229997 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:35.230010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.232894 (+ 2884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.232900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:35.234549 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:35.234561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.236341 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.236347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:35.236518 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.236542 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:52:35.236651 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:52:35.236783 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8163,"thread_start_us":8031,"threads_started":1}
I20260925 02:52:35.240797 14704 tablet_service.cc:1467] Processing CreateTablet for tablet 523c2dcc0aae489a9ecb5c36fcd5880c (DEFAULT_TABLE table=test-table [id=6e0482fcdace4a64a0e988059e6a0a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:35.242206 14704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523c2dcc0aae489a9ecb5c36fcd5880c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:35.250514 14790 tablet_bootstrap.cc:492] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2: Bootstrap starting.
I20260925 02:52:35.252607 14792 tablet_bootstrap.cc:492] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d: Bootstrap starting.
I20260925 02:52:35.254655 14790 tablet_bootstrap.cc:654] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:35.254808 14793 tablet_bootstrap.cc:492] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684: Bootstrap starting.
I20260925 02:52:35.256057 14792 tablet_bootstrap.cc:654] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:35.256069 14790 log.cc:824] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:35.257305 14792 log.cc:824] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:35.258529 14793 tablet_bootstrap.cc:654] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:35.260123 14793 log.cc:824] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:35.263484 14790 tablet_bootstrap.cc:492] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2: No bootstrap required, opened a new log
I20260925 02:52:35.263731 14793 tablet_bootstrap.cc:492] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684: No bootstrap required, opened a new log
I20260925 02:52:35.263827 14792 tablet_bootstrap.cc:492] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d: No bootstrap required, opened a new log
I20260925 02:52:35.263855 14790 ts_tablet_manager.cc:1397] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260925 02:52:35.264006 14793 ts_tablet_manager.cc:1397] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 02:52:35.264067 14792 ts_tablet_manager.cc:1397] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260925 02:52:35.272002 14790 raft_consensus.cc:348] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e454dc0e1e4d4806972f696ab589b8e2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39279 } } peers { permanent_uuid: "f44be1dd515b4815bf0b8d8a2c87522d" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35209 } } peers { permanent_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35653 } }
I20260925 02:52:35.272493 14790 raft_consensus.cc:374] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:35.272627 14790 raft_consensus.cc:729] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e454dc0e1e4d4806972f696ab589b8e2, State: Initialized, Role: FOLLOWER
I20260925 02:52:35.273339 14790 consensus_queue.cc:260] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e454dc0e1e4d4806972f696ab589b8e2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39279 } } peers { permanent_uuid: "f44be1dd515b4815bf0b8d8a2c87522d" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35209 } } peers { permanent_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35653 } }
I20260925 02:52:35.274868 14792 raft_consensus.cc:348] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e454dc0e1e4d4806972f696ab589b8e2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39279 } } peers { permanent_uuid: "f44be1dd515b4815bf0b8d8a2c87522d" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35209 } } peers { permanent_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35653 } }
I20260925 02:52:35.274895 14793 raft_consensus.cc:348] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e454dc0e1e4d4806972f696ab589b8e2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39279 } } peers { permanent_uuid: "f44be1dd515b4815bf0b8d8a2c87522d" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35209 } } peers { permanent_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35653 } }
I20260925 02:52:35.275315 14793 raft_consensus.cc:374] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:35.275331 14790 ts_tablet_manager.cc:1428] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 02:52:35.275345 14792 raft_consensus.cc:374] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:35.275410 14793 raft_consensus.cc:729] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1caf8ad5eef4dd4acb1fcd953203684, State: Initialized, Role: FOLLOWER
I20260925 02:52:35.275452 14792 raft_consensus.cc:729] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44be1dd515b4815bf0b8d8a2c87522d, State: Initialized, Role: FOLLOWER
I20260925 02:52:35.276108 14792 consensus_queue.cc:260] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e454dc0e1e4d4806972f696ab589b8e2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39279 } } peers { permanent_uuid: "f44be1dd515b4815bf0b8d8a2c87522d" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35209 } } peers { permanent_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35653 } }
I20260925 02:52:35.276060 14793 consensus_queue.cc:260] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e454dc0e1e4d4806972f696ab589b8e2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39279 } } peers { permanent_uuid: "f44be1dd515b4815bf0b8d8a2c87522d" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35209 } } peers { permanent_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35653 } }
I20260925 02:52:35.280345 14635 heartbeater.cc:503] Master 127.2.172.126:43509 was elected leader, sending a full tablet report...
I20260925 02:52:35.282816 14792 ts_tablet_manager.cc:1428] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d: Time spent starting tablet: real 0.019s user 0.016s sys 0.000s
I20260925 02:52:35.283188 14771 heartbeater.cc:503] Master 127.2.172.126:43509 was elected leader, sending a full tablet report...
I20260925 02:52:35.284056 14793 ts_tablet_manager.cc:1428] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
W20260925 02:52:35.313575 14636 tablet.cc:2369] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:52:35.364444 14798 raft_consensus.cc:482] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:52:35.364725 14798 raft_consensus.cc:504] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e454dc0e1e4d4806972f696ab589b8e2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39279 } } peers { permanent_uuid: "f44be1dd515b4815bf0b8d8a2c87522d" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35209 } } peers { permanent_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35653 } }
I20260925 02:52:35.366413 14798 leader_election.cc:288] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e454dc0e1e4d4806972f696ab589b8e2 (127.2.172.65:39279), f44be1dd515b4815bf0b8d8a2c87522d (127.2.172.66:35209)
I20260925 02:52:35.373322 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.367264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:53867 (local address 127.2.172.66:35209)
0925 02:52:35.367501 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:35.367507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:35.367529 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:52:35.367621 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:35.367627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:35.367718 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:35.367819 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:35.368243 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.369267 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.371427 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.372332 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.372491 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.372564 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:35.372900 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 02:52:35.373054 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:52:35.373339 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.366684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:35209 (local address 127.2.172.67:53867)
0925 02:52:35.366910 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:35.366936 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:35.367071 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:35.367855 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:35.367862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:35.367894 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:35.368101 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:35.368112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.369435 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.369440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:35.371237 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:35.371249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.372482 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.372488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:35.372693 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.372736 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 02:52:35.372855 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 02:52:35.372997 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":1}
I20260925 02:52:35.373828 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.366849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:39279 (local address 127.2.172.67:55675)
0925 02:52:35.367636 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:35.367652 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:35.367773 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:35.368116 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:35.368119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:35.368137 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:35.368542 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:35.368555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.369745 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.369748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:35.371528 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:35.371543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.372838 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.372846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:35.373005 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.373032 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:52:35.373121 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:52:35.373230 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":424,"threads_started":1}
I20260925 02:52:35.373931 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.366816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:55675 (local address 127.2.172.65:39279)
0925 02:52:35.367022 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:35.367027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:35.367741 (+ 714us) server_negotiation.cc:407] Connection header received
0925 02:52:35.367901 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:35.367906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:35.367978 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:35.368086 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:35.368750 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.369578 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.371718 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.372667 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.372793 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.372885 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:35.373147 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 02:52:35.373277 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:52:35.374401 14589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523c2dcc0aae489a9ecb5c36fcd5880c" candidate_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44be1dd515b4815bf0b8d8a2c87522d" is_pre_election: true
I20260925 02:52:35.374926 14589 raft_consensus.cc:2454] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1caf8ad5eef4dd4acb1fcd953203684 in term 0.
I20260925 02:52:35.374868 14444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523c2dcc0aae489a9ecb5c36fcd5880c" candidate_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e454dc0e1e4d4806972f696ab589b8e2" is_pre_election: true
I20260925 02:52:35.375452 14444 raft_consensus.cc:2454] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1caf8ad5eef4dd4acb1fcd953203684 in term 0.
I20260925 02:52:35.375805 14657 leader_election.cc:302] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1caf8ad5eef4dd4acb1fcd953203684, f44be1dd515b4815bf0b8d8a2c87522d; no voters:
I20260925 02:52:35.376240 14798 raft_consensus.cc:2781] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:52:35.376436 14798 raft_consensus.cc:482] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:52:35.376554 14798 raft_consensus.cc:3037] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:35.378753 14798 raft_consensus.cc:504] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e454dc0e1e4d4806972f696ab589b8e2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39279 } } peers { permanent_uuid: "f44be1dd515b4815bf0b8d8a2c87522d" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35209 } } peers { permanent_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35653 } }
I20260925 02:52:35.379333 14798 leader_election.cc:288] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [CANDIDATE]: Term 1 election: Requested vote from peers e454dc0e1e4d4806972f696ab589b8e2 (127.2.172.65:39279), f44be1dd515b4815bf0b8d8a2c87522d (127.2.172.66:35209)
I20260925 02:52:35.379822 14444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523c2dcc0aae489a9ecb5c36fcd5880c" candidate_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e454dc0e1e4d4806972f696ab589b8e2"
I20260925 02:52:35.379878 14589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523c2dcc0aae489a9ecb5c36fcd5880c" candidate_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44be1dd515b4815bf0b8d8a2c87522d"
I20260925 02:52:35.380048 14444 raft_consensus.cc:3037] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:35.380067 14589 raft_consensus.cc:3037] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:35.383320 14444 raft_consensus.cc:2454] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1caf8ad5eef4dd4acb1fcd953203684 in term 1.
I20260925 02:52:35.383312 14589 raft_consensus.cc:2454] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1caf8ad5eef4dd4acb1fcd953203684 in term 1.
I20260925 02:52:35.383774 14657 leader_election.cc:302] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1caf8ad5eef4dd4acb1fcd953203684, f44be1dd515b4815bf0b8d8a2c87522d; no voters:
I20260925 02:52:35.384079 14798 raft_consensus.cc:2781] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:35.384634 14798 raft_consensus.cc:686] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 1 LEADER]: Becoming Leader. State: Replica: d1caf8ad5eef4dd4acb1fcd953203684, State: Running, Role: LEADER
I20260925 02:52:35.385187 14798 consensus_queue.cc:237] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e454dc0e1e4d4806972f696ab589b8e2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39279 } } peers { permanent_uuid: "f44be1dd515b4815bf0b8d8a2c87522d" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35209 } } peers { permanent_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35653 } }
W20260925 02:52:35.388798 14772 tablet.cc:2369] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:52:35.390013 14273 catalog_manager.cc:5495] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1caf8ad5eef4dd4acb1fcd953203684 (127.2.172.67). New cstate: current_term: 1 leader_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e454dc0e1e4d4806972f696ab589b8e2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f44be1dd515b4815bf0b8d8a2c87522d" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1caf8ad5eef4dd4acb1fcd953203684" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35653 } health_report { overall_health: HEALTHY } } }
I20260925 02:52:35.420526 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.415356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.2.172.67:35653)
0925 02:52:35.415535 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:35.415539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:35.415598 (+ 59us) server_negotiation.cc:407] Connection header received
0925 02:52:35.415692 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:35.415696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:35.415773 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:35.415903 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:35.416246 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.416977 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.418446 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.418854 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.418945 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.419406 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:35.419998 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:35.420120 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:35.420247 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 02:52:35.420311 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:52:35.429299 14444 raft_consensus.cc:1261] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2 [term 1 FOLLOWER]: Refusing update from remote peer d1caf8ad5eef4dd4acb1fcd953203684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:35.429322 14589 raft_consensus.cc:1261] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d [term 1 FOLLOWER]: Refusing update from remote peer d1caf8ad5eef4dd4acb1fcd953203684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:35.430383 14801 consensus_queue.cc:1035] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e454dc0e1e4d4806972f696ab589b8e2" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 39279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:35.430824 14798 consensus_queue.cc:1035] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f44be1dd515b4815bf0b8d8a2c87522d" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:35.460377 14272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:35.460714 14272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:35.464982 14272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d213da55e0747b39be74207b2decf09: Sending DeleteTablet for 3 replicas of tablet 523c2dcc0aae489a9ecb5c36fcd5880c
I20260925 02:52:35.466471 14424 tablet_service.cc:1514] Processing DeleteTablet for tablet 523c2dcc0aae489a9ecb5c36fcd5880c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:35 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:45836
I20260925 02:52:35.466746 14569 tablet_service.cc:1514] Processing DeleteTablet for tablet 523c2dcc0aae489a9ecb5c36fcd5880c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:35 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:45362
I20260925 02:52:35.466826 14704 tablet_service.cc:1514] Processing DeleteTablet for tablet 523c2dcc0aae489a9ecb5c36fcd5880c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:35 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:41874
I20260925 02:52:35.471472 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 14355
I20260925 02:52:35.478811 14816 tablet_replica.cc:330] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684: stopping tablet replica
I20260925 02:52:35.479590 14816 raft_consensus.cc:2229] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:52:35.480535 14816 raft_consensus.cc:2258] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:35.481225 14815 tablet_replica.cc:330] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d: stopping tablet replica
I20260925 02:52:35.481889 14815 raft_consensus.cc:2229] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:35.484495 14815 raft_consensus.cc:2258] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:35.486692 14816 ts_tablet_manager.cc:1905] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:35.488211 14815 ts_tablet_manager.cc:1905] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:35.494652 14816 ts_tablet_manager.cc:1918] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:52:35.494964 14816 log.cc:1196] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/523c2dcc0aae489a9ecb5c36fcd5880c
I20260925 02:52:35.495645 14816 ts_tablet_manager.cc:1939] T 523c2dcc0aae489a9ecb5c36fcd5880c P d1caf8ad5eef4dd4acb1fcd953203684: Deleting consensus metadata
I20260925 02:52:35.497779 14257 catalog_manager.cc:4841] TS d1caf8ad5eef4dd4acb1fcd953203684 (127.2.172.67:35653): tablet 523c2dcc0aae489a9ecb5c36fcd5880c (table test-table [id=6e0482fcdace4a64a0e988059e6a0a5d]) successfully deleted
I20260925 02:52:35.498894 14814 tablet_replica.cc:330] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2: stopping tablet replica
I20260925 02:52:35.498160 14815 ts_tablet_manager.cc:1918] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:52:35.499301 14815 log.cc:1196] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/523c2dcc0aae489a9ecb5c36fcd5880c
I20260925 02:52:35.499430 14814 raft_consensus.cc:2229] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:35.500000 14815 ts_tablet_manager.cc:1939] T 523c2dcc0aae489a9ecb5c36fcd5880c P f44be1dd515b4815bf0b8d8a2c87522d: Deleting consensus metadata
I20260925 02:52:35.500048 14814 raft_consensus.cc:2258] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:35.502137 14814 ts_tablet_manager.cc:1905] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:35.502277 14257 catalog_manager.cc:4841] TS f44be1dd515b4815bf0b8d8a2c87522d (127.2.172.66:35209): tablet 523c2dcc0aae489a9ecb5c36fcd5880c (table test-table [id=6e0482fcdace4a64a0e988059e6a0a5d]) successfully deleted
Sep 25 02:52:35 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:52:35.508677 14814 ts_tablet_manager.cc:1918] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:52:35.508881 14814 log.cc:1196] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/523c2dcc0aae489a9ecb5c36fcd5880c
I20260925 02:52:35.509471 14814 ts_tablet_manager.cc:1939] T 523c2dcc0aae489a9ecb5c36fcd5880c P e454dc0e1e4d4806972f696ab589b8e2: Deleting consensus metadata
I20260925 02:52:35.511533 14258 catalog_manager.cc:4841] TS e454dc0e1e4d4806972f696ab589b8e2 (127.2.172.65:39279): tablet 523c2dcc0aae489a9ecb5c36fcd5880c (table test-table [id=6e0482fcdace4a64a0e988059e6a0a5d]) successfully deleted
I20260925 02:52:35.515511 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.473447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.2.172.65:39279)
0925 02:52:35.473715 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:35.473724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:35.491997 (+ 18273us) server_negotiation.cc:407] Connection header received
0925 02:52:35.492316 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:35.492323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:35.492416 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:35.492498 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:35.498293 (+ 5795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.499536 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.501574 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.502237 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.502336 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.508494 (+ 6158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:35.508511 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:35.508530 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:35.508577 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:35.512566 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:35.513515 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:35.513522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:35.513525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:35.513652 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:35.514094 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:35.514097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:35.514099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:35.514444 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:35.514658 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:35.515020 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 02:52:35.515205 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:52:35.516142 14485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:52:35.666388 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 14355
I20260925 02:52:35.685151 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 14496
Sep 25 02:52:35 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:52:35.703289 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.686066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.2.172.66:35209)
0925 02:52:35.686282 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:35.686289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:35.686312 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:52:35.686391 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:35.686398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:35.686482 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:35.686596 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:35.688063 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.689274 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.690696 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.691325 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.691464 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.696325 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:35.696344 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:35.696365 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:35.696415 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:35.700397 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:35.701111 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:35.701121 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:35.701125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:35.701281 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:35.701645 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:35.701649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:35.701653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:35.702126 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:35.702427 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:35.702673 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 02:52:35.702900 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:52:35.704147 14630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:52:35.796650 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 14496
I20260925 02:52:35.820082 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 14641
Sep 25 02:52:35 dist-test-slave-g1xh krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:52:35.839686 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:35.820871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.2.172.67:35653)
0925 02:52:35.821046 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:35.821050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:35.821068 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:52:35.821291 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:35.821295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:35.821391 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:35.821489 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:35.823342 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.824399 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.826403 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:35.826920 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:35.827037 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:35.833032 (+ 5995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:35.833051 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:35.833067 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:35.833111 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:35.836521 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:35.837477 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:35.837485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:35.837487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:35.837644 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:35.838088 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:35.838091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:35.838093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:35.838484 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:35.838737 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:35.839143 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 02:52:35.839345 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:52:35.840821 14766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:52:35.967685 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 14641
I20260925 02:52:35.987599 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 14241
I20260925 02:52:35.988847 14303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:52:36.100740 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 14241
I20260925 02:52:36.119669 2737 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:52:36.456233 2737 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:52:36.456571 2737 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:52:36.803623 2737 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:52:36.804 UTC [13824] LOG: received smart shutdown request
2026-09-25 02:52:36.808 UTC [13824] LOG: background worker "logical replication launcher" (PID 13833) exited with exit code 1
2026-09-25 02:52:36.808 UTC [13828] LOG: shutting down
2026-09-25 02:52:37.001 UTC [13824] LOG: database system is shut down
2026-09-25T02:52:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54205 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:37 dist-test-slave-g1xh krb5kdc[14830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:37 dist-test-slave-g1xh krb5kdc[14830](info): set up 2 sockets
Sep 25 02:52:37 dist-test-slave-g1xh krb5kdc[14830](info): commencing operation
krb5kdc: starting...
W20260925 02:52:38.293924 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.005s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:52:38 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/rangeradmin_127.2.172.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/rangeradmin_127.2.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/rangerlookup_127.2.172.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/rangerlookup_127.2.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/HTTP_127.2.172.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/HTTP_127.2.172.84@KRBTEST.COM.keytab.
I20260925 02:52:38.667164 2737 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task450sZB/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/postgres -l logfile start
2026-09-25 02:52:41.367 UTC [14860] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:52:41.367 UTC [14860] LOG: listening on IPv4 address "127.2.172.84", port 39327
2026-09-25 02:52:41.372 UTC [14860] LOG: listening on Unix socket "/tmp/.s.PGSQL.39327"
2026-09-25 02:52:41.402 UTC [14863] LOG: database system was shut down at 2026-09-25 02:52:39 UTC
2026-09-25 02:52:41.406 UTC [14860] LOG: database system is ready to accept connections
I20260925 02:52:42.429792 2737 mini_postgres.cc:96] Postgres bound to 39327
2026-09-25 02:52:42.450 UTC [14871] FATAL: database "slave" does not exist
127.2.172.84:39327 - accepting connections
I20260925 02:52:42.451200 2737 mini_ranger.cc:161] Starting Ranger...
I20260925 02:52:42.480352 2737 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:52:43.035955 2737 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:52:43.036069 2737 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-admin
2026-09-25 02:52:43,342 [I] DB FLAVOR :POSTGRES
2026-09-25 02:52:43,343 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:52:43,343 [I] Checking connection..
2026-09-25 02:52:43,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:52:43,609 [I] Checking connection passed.
2026-09-25 02:52:43,609 [I] --------- Verifying version history table ---------
2026-09-25 02:52:43,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:52:43,848 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:52:43,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:52:44,088 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:52:44,088 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:52:44,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:52:44.344 UTC [14938] WARNING: there is no transaction in progress
2026-09-25 02:52:44,351 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:52:44,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:52:44,599 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:52:44,599 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:52:44,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:52:44,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:52:45,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g1xh.c.gcp-upstream.internal','N') ;"
2026-09-25 02:52:45,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:52:45,639 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:52:45,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:52:45,875 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:52:45,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:52:46,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:52:46,390 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:52:46,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:52:47.448 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.467 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.487 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.506 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.526 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.548 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.619 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.627 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.644 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.655 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.667 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.683 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.694 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.702 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.712 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:47.721 UTC [15046] WARNING: there is no transaction in progress
2026-09-25 02:52:48,503 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:52:48,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:52:48,744 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:52:48,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:52:49,006 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:52:49,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:52:49,256 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:52:49,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:52:49,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:52:49,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g1xh.c.gcp-upstream.internal';"
2026-09-25 02:52:50,004 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:52:50,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:52:50,276 [I] DB_PATCHES have already been applied
I20260925 02:52:50.282020 2737 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-admin:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task450sZB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:52:50.282117 2737 mini_ranger.cc:193] Using host: 127.2.172.84
I20260925 02:52:50.299145 2737 mini_ranger.cc:227] Ranger admin URL: http://127.2.172.84:47147
Sep 25, 2026 2:52:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:52:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:52:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:52:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:52:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:52:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:52:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:52:51.459117 2737 mini_ranger.cc:160] Time spent starting Ranger: real 9.008s user 0.000s sys 0.049s
Sep 25, 2026 2:53:11 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:53:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 2:53:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:53:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:53:11 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:53:12 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:53:13.271011 2737 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/rangerkms_127.2.172.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/rangerkms_127.2.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.172.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.172.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.172.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/HTTP_127.2.172.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.172.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/HTTP_127.2.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:53:13 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:53:13.453181 2737 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:53:13.476647 2737 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:53:14.467197 2737 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:53:14.467335 2737 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-kms
I20260925 02:53:14.467398 2737 mini_ranger_kms.cc:227] Using postgres at 127.2.172.84:39327
2026-09-25 02:53:14,839 [I] DB FLAVOR :POSTGRES
2026-09-25 02:53:14,839 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:53:14,839 [I] Checking connection
2026-09-25 02:53:14,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:53:15,098 [I] connection success
2026-09-25 02:53:15,098 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:53:15,098 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:53:15,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:53:15,389 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:53:15,389 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:53:15,389 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:53:15,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.172.84:39327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:53:15,652 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:53:16.146971 2737 mini_ranger_kms.cc:310] Created kms service
I20260925 02:53:16.365890 2737 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:53:16.413354 2737 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:53:16 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:53:16.838574 2737 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:53:16.838920 2737 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-kms:/tmp/dist-test-task450sZB/build/asan/bin/postgresql.jar:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task450sZB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:53:16.838986 2737 mini_ranger_kms.cc:240] Using host: 127.2.172.84
I20260925 02:53:16.858448 2737 mini_ranger_kms.cc:276] Ranger KMS PID: 15228
I20260925 02:53:16.858590 2737 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.172.84:48717
Sep 25, 2026 2:53:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:53:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:53:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:53:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:53:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:53:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 02:53:19.184404 2737 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.731s user 0.083s sys 0.313s
I20260925 02:53:19.184677 2737 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:53:19.184861 2737 mini_ranger_kms.cc:401] 127.2.172.84:48717/kms/v1/keys
Sep 25 02:53:19 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304799, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.172.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 02:53:19 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304799, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.172.84@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
Sep 25 02:53:20 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304799, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.172.84@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
Sep 25 02:53:22 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:53:22.611943 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:46151
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:46151
--ranger_config_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.172.84:48717/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:22.953454 15280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:22.953747 15280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:22.953926 15280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:22.961241 15280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:22.961350 15280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:22.961464 15280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:22.961570 15280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:22.961642 15280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:22.971974 15280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.172.84:48717/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:46151
--ranger_config_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:46151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:22.973287 15280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:22.974643 15280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:22.986450 15288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:22.986577 15285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:22.986636 15286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:22.987803 15280 server_base.cc:1034] running on GCE node
I20260925 02:53:22.988629 15280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:22.990023 15280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:22.991291 15280 hybrid_clock.cc:648] HybridClock initialized: now 1790304802991263 us; error 56 us; skew 500 ppm
I20260925 02:53:22.991761 15280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:22 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304802, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:22.994817 15280 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:53:22.996385 15280 webserver.cc:460] Webserver started at http://127.2.172.126:46261/ using document root <none> and password file <none>
I20260925 02:53:22.997007 15280 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:22.997107 15280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:22.997421 15280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:53:22 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304802, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:53:23.060184 15280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "38df122079a248b7b01be87495536928"
format_stamp: "Formatted at 2026-09-25 02:53:23 on dist-test-slave-g1xh"
server_key: "1d515be175c6c5498278a6210f3df7e7"
server_key_iv: "c6f9b2507d08961c9d66f0df04ea93cf"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:23.061079 15280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "38df122079a248b7b01be87495536928"
format_stamp: "Formatted at 2026-09-25 02:53:23 on dist-test-slave-g1xh"
server_key: "1d515be175c6c5498278a6210f3df7e7"
server_key_iv: "c6f9b2507d08961c9d66f0df04ea93cf"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:23.066629 15280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:53:23.086568 15295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:23.088275 15280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:53:23.088475 15280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "38df122079a248b7b01be87495536928"
format_stamp: "Formatted at 2026-09-25 02:53:23 on dist-test-slave-g1xh"
server_key: "1d515be175c6c5498278a6210f3df7e7"
server_key_iv: "c6f9b2507d08961c9d66f0df04ea93cf"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:23.088722 15280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:23.114780 15280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:23.119004 15280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:23.119355 15280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:23.139330 15280 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:46151
I20260925 02:53:23.139330 15347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:46151 every 8 connection(s)
I20260925 02:53:23.140828 15280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:23.144814 15348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:23.149263 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 15280
I20260925 02:53:23.149478 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:53:23.150321 2737 external_mini_cluster.cc:1428] Setting key 377b71cb5fecef63a8528c0b2517ddcd
I20260925 02:53:23.159000 15348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928: Bootstrap starting.
I20260925 02:53:23.164399 15348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:23.166860 15348 log.cc:824] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:23.170642 15348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928: No bootstrap required, opened a new log
I20260925 02:53:23.180012 15348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38df122079a248b7b01be87495536928" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46151 } }
I20260925 02:53:23.180655 15348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:23.180795 15348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38df122079a248b7b01be87495536928, State: Initialized, Role: FOLLOWER
Sep 25 02:53:23 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:23.181452 15348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38df122079a248b7b01be87495536928" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46151 } }
I20260925 02:53:23.181715 15348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:23.181892 15348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:23.182081 15348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:23.184870 15348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38df122079a248b7b01be87495536928" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46151 } }
I20260925 02:53:23.185465 15348 leader_election.cc:302] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38df122079a248b7b01be87495536928; no voters:
I20260925 02:53:23.186230 15348 leader_election.cc:288] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:23.186388 15353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:23.190631 15353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 1 LEADER]: Becoming Leader. State: Replica: 38df122079a248b7b01be87495536928, State: Running, Role: LEADER
I20260925 02:53:23.191521 15353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38df122079a248b7b01be87495536928" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46151 } }
I20260925 02:53:23.193096 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:23.152485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.2.172.126:46151)
0925 02:53:23.165263 (+ 12778us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:23.165287 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:23.166558 (+ 1271us) server_negotiation.cc:407] Connection header received
0925 02:53:23.169086 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:23.169116 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:23.169619 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:23.170077 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:23.171685 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:23.173797 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:23.175786 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:23.176972 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:23.177209 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:23.182413 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:23.182424 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:23.182445 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:23.182500 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:23.186665 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:23.187637 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:23.187645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:23.187655 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:23.187831 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:23.188285 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:23.188289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:23.188292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:23.189041 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:23.190475 (+ 1434us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:23.190776 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 02:53:23.191222 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12432,"thread_start_us":13095,"threads_started":1}
I20260925 02:53:23.193838 15348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:23.201244 15355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38df122079a248b7b01be87495536928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38df122079a248b7b01be87495536928" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46151 } } }
I20260925 02:53:23.201769 15355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:23.202626 15354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38df122079a248b7b01be87495536928. Latest consensus state: current_term: 1 leader_uuid: "38df122079a248b7b01be87495536928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38df122079a248b7b01be87495536928" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46151 } } }
I20260925 02:53:23.203109 15354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:23.210932 15359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:23.222222 15359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:23.234710 15359 catalog_manager.cc:1347] Generated new cluster ID: 1c36a996eed946bf8ca570fd26038eff
I20260925 02:53:23.234834 15359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:23.253520 15359 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:23.255622 15359 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:23.263968 15359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928: Generated new TSK 0
I20260925 02:53:23.265165 15359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:53:23.267668 15348 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:53:24.054952 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.172.84:48717/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:46151
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:53:24 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 02:53:24.506824 15394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:24.507226 15394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:24.507442 15394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:24.519214 15394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:24.519371 15394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:24.519670 15394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:53:24.537259 15394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.172.84:48717/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:46151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:24.539196 15394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:24.541330 15394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:24.560478 15405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:24.564821 15404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:24.566468 15407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:24.567173 15394 server_base.cc:1034] running on GCE node
I20260925 02:53:24.568046 15394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:24.569386 15394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:24.570744 15394 hybrid_clock.cc:648] HybridClock initialized: now 1790304804570633 us; error 93 us; skew 500 ppm
I20260925 02:53:24.571182 15394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:24 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:24.576040 15394 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:53:24.581334 15394 webserver.cc:460] Webserver started at http://127.2.172.65:43451/ using document root <none> and password file <none>
I20260925 02:53:24.582412 15394 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:24.582592 15394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:24.582988 15394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:53:24 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:53:24.620725 15394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "f23c39320d4f4d66a22ae59bdcbe5700"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-g1xh"
server_key: "74d3f7042c3e552bdf311eeb4b027f70"
server_key_iv: "b4b3ecf6e9af16297c21493e2ade6d51"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:24.625190 15394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "f23c39320d4f4d66a22ae59bdcbe5700"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-g1xh"
server_key: "74d3f7042c3e552bdf311eeb4b027f70"
server_key_iv: "b4b3ecf6e9af16297c21493e2ade6d51"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:24.633985 15394 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260925 02:53:24.659284 15414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.662966 15394 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260925 02:53:24.663234 15394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "f23c39320d4f4d66a22ae59bdcbe5700"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-g1xh"
server_key: "74d3f7042c3e552bdf311eeb4b027f70"
server_key_iv: "b4b3ecf6e9af16297c21493e2ade6d51"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:24.663570 15394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:24.692077 15394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:24.697587 15394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:24.697940 15394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:24.699230 15394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:24.701097 15394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:24.701196 15394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.701305 15394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:24.701365 15394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.766963 15394 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:45769
I20260925 02:53:24.767004 15527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:45769 every 8 connection(s)
I20260925 02:53:24.768724 15394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:24.772212 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 15394
I20260925 02:53:24.772444 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:24.773231 2737 external_mini_cluster.cc:1428] Setting key 5ef9dd2e06147f01f51b34c16128555a
Sep 25 02:53:24 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
I20260925 02:53:24.827427 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:24.791152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:59611 (local address 127.2.172.126:46151)
0925 02:53:24.804005 (+ 12853us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:24.804014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:24.804721 (+ 707us) server_negotiation.cc:407] Connection header received
0925 02:53:24.807775 (+ 3054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:24.807782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:24.807899 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:24.807989 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:24.810321 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.811465 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.813985 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.814628 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.814723 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:24.819724 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:24.819733 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:24.819738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:24.819784 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:24.823596 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:24.824620 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:24.824627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:24.824630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:24.824780 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:24.825316 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:24.825321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:24.825324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:24.825682 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:24.825810 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:24.826809 (+ 999us) server_negotiation.cc:299] Negotiation successful
0925 02:53:24.827046 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12662,"thread_start_us":12538,"threads_started":1}
I20260925 02:53:24.830113 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:24.790980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:46151 (local address 127.2.172.65:59611)
0925 02:53:24.804492 (+ 13512us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:24.804571 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:24.807211 (+ 2640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:24.808591 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:24.808604 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:24.809243 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:24.810104 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:24.810131 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.811723 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.811729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:24.813814 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:24.813830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.815067 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.815075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:24.815375 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:24.816267 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:24.816301 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:24.819357 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:24.823791 (+ 4434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:24.823801 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:24.823822 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:24.824443 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:24.824925 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:24.824929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:24.824933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:24.825188 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:24.825904 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:24.825913 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:24.826359 (+ 446us) client_negotiation.cc:769] Sending connection context
0925 02:53:24.828046 (+ 1687us) client_negotiation.cc:240] Negotiation successful
0925 02:53:24.828652 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13131,"thread_start_us":16474,"threads_started":1}
I20260925 02:53:24.832424 15528 heartbeater.cc:347] Connected to a master server at 127.2.172.126:46151
I20260925 02:53:24.832782 15528 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:24.833640 15528 heartbeater.cc:511] Master 127.2.172.126:46151 requested a full tablet report, sending...
I20260925 02:53:24.836277 15313 ts_manager.cc:194] Registered new tserver with Master: f23c39320d4f4d66a22ae59bdcbe5700 (127.2.172.65:45769)
I20260925 02:53:24.838361 15313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:59611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:53:24.869570 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.172.84:48717/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:46151
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:53:24 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
Sep 25 02:53:25 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
W20260925 02:53:25.326493 15534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:25.327004 15534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:25.327190 15534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:25.337960 15534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:25.338181 15534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:25.338383 15534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:53:25.354221 15534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.172.84:48717/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:46151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:25.356356 15534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:25.358274 15534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:25.386142 15542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:25.392153 15541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:25.395929 15544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:25.397109 15534 server_base.cc:1034] running on GCE node
I20260925 02:53:25.397991 15534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:25.399147 15534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:25.400463 15534 hybrid_clock.cc:648] HybridClock initialized: now 1790304805400341 us; error 110 us; skew 500 ppm
I20260925 02:53:25.400831 15534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:25 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304805, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:25.404908 15534 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:53:25.430176 15534 webserver.cc:460] Webserver started at http://127.2.172.66:36283/ using document root <none> and password file <none>
I20260925 02:53:25.431401 15534 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:25.431687 15534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:25.432149 15534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:53:25 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304805, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:53:25.470408 15534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "3337e40d25924f899b39b9ecee5d41ab"
format_stamp: "Formatted at 2026-09-25 02:53:25 on dist-test-slave-g1xh"
server_key: "402ab7c05832d30d081a7bf14b8e7f59"
server_key_iv: "bce5c2004eae655db4646c42575e9124"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:25.471509 15534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "3337e40d25924f899b39b9ecee5d41ab"
format_stamp: "Formatted at 2026-09-25 02:53:25 on dist-test-slave-g1xh"
server_key: "402ab7c05832d30d081a7bf14b8e7f59"
server_key_iv: "bce5c2004eae655db4646c42575e9124"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:25.477422 15534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:53:25.519516 15552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:25.522009 15534 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.004s
I20260925 02:53:25.522336 15534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "3337e40d25924f899b39b9ecee5d41ab"
format_stamp: "Formatted at 2026-09-25 02:53:25 on dist-test-slave-g1xh"
server_key: "402ab7c05832d30d081a7bf14b8e7f59"
server_key_iv: "bce5c2004eae655db4646c42575e9124"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:25.522702 15534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:25.571399 15534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:25.577178 15534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:25.577527 15534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:25.578789 15534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:25.580765 15534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:25.580899 15534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 02:53:25.581049 15534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:25.581117 15534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:25.628798 15534 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:39427
I20260925 02:53:25.628810 15671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:39427 every 8 connection(s)
I20260925 02:53:25.630692 15534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:25.632068 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 15534
I20260925 02:53:25.632247 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:53:25.632648 2737 external_mini_cluster.cc:1428] Setting key 6a009dea7218f927223051db61a45573
Sep 25 02:53:25 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304805, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
I20260925 02:53:25.671029 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:25.635746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:33227 (local address 127.2.172.126:46151)
0925 02:53:25.649137 (+ 13391us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:25.649146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:25.649172 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:53:25.650560 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:25.650566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:25.650662 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:25.651667 (+ 1005us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:25.653000 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.654246 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.657048 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.657738 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.657881 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.662839 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:25.662849 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:25.662858 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:25.662911 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:25.666934 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:25.668021 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:25.668029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:25.668038 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:25.668257 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:25.668791 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:25.668798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:25.668803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:25.669176 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:25.669373 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:25.670512 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0925 02:53:25.670758 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13201,"thread_start_us":13073,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
I20260925 02:53:25.672748 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:25.635770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:46151 (local address 127.2.172.66:33227)
0925 02:53:25.648201 (+ 12431us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:25.648273 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:25.650250 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:25.651116 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:25.651130 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:25.651871 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:25.652763 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.652792 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.654458 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.654466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.656858 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.656878 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.658022 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.658031 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.658392 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.659289 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:25.659325 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:25.662504 (+ 3179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:25.667135 (+ 4631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:25.667145 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:25.667165 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:25.667856 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:25.668389 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:25.668394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:25.668398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:25.668671 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:25.669367 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:25.669378 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:25.669958 (+ 580us) client_negotiation.cc:769] Sending connection context
0925 02:53:25.670381 (+ 423us) client_negotiation.cc:240] Negotiation successful
0925 02:53:25.671087 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12096,"thread_start_us":11836,"threads_started":1}
I20260925 02:53:25.680281 15672 heartbeater.cc:347] Connected to a master server at 127.2.172.126:46151
I20260925 02:53:25.680615 15672 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:25.681501 15672 heartbeater.cc:511] Master 127.2.172.126:46151 requested a full tablet report, sending...
I20260925 02:53:25.683400 15313 ts_manager.cc:194] Registered new tserver with Master: 3337e40d25924f899b39b9ecee5d41ab (127.2.172.66:39427)
I20260925 02:53:25.684791 15313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:33227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:53:25.703825 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.172.84:48717/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:46151
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:53:25.842346 15528 heartbeater.cc:503] Master 127.2.172.126:46151 was elected leader, sending a full tablet report...
W20260925 02:53:26.052924 15678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:26.053249 15678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:26.053416 15678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:26.060855 15678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:26.061054 15678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:26.061233 15678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:53:26.071444 15678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.172.84:48717/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:46151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:26.072908 15678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:26.074210 15678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:26.090242 15684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:26.090407 15686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:26.090660 15683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:26.091207 15678 server_base.cc:1034] running on GCE node
I20260925 02:53:26.091969 15678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:26.093168 15678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:26.094421 15678 hybrid_clock.cc:648] HybridClock initialized: now 1790304806094370 us; error 38 us; skew 500 ppm
I20260925 02:53:26.094828 15678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:26 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304806, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:26.098404 15678 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:53:26.100219 15678 webserver.cc:460] Webserver started at http://127.2.172.67:33209/ using document root <none> and password file <none>
I20260925 02:53:26.100931 15678 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:26.101085 15678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:26.101408 15678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:53:26 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304806, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:53:26.122788 15678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "1931d369be0d4d53bde9574bdeda1cbe"
format_stamp: "Formatted at 2026-09-25 02:53:26 on dist-test-slave-g1xh"
server_key: "b4e633f364e9629cad397280e4adaa42"
server_key_iv: "95d76ca13c66a617cbc2088d9427f95c"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:26.124573 15678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "1931d369be0d4d53bde9574bdeda1cbe"
format_stamp: "Formatted at 2026-09-25 02:53:26 on dist-test-slave-g1xh"
server_key: "b4e633f364e9629cad397280e4adaa42"
server_key_iv: "95d76ca13c66a617cbc2088d9427f95c"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:26.130784 15678 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:53:26.151937 15693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:26.153538 15678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:53:26.153745 15678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "1931d369be0d4d53bde9574bdeda1cbe"
format_stamp: "Formatted at 2026-09-25 02:53:26 on dist-test-slave-g1xh"
server_key: "b4e633f364e9629cad397280e4adaa42"
server_key_iv: "95d76ca13c66a617cbc2088d9427f95c"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:26.153995 15678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:26.167737 15678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:26.172571 15678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:26.173027 15678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:26.174396 15678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:26.176339 15678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:26.176446 15678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:26.176575 15678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:26.176648 15678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:26.216512 15678 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:34809
I20260925 02:53:26.216584 15807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:34809 every 8 connection(s)
I20260925 02:53:26.217849 15678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:26.223706 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 15678
I20260925 02:53:26.223897 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:26.224515 2737 external_mini_cluster.cc:1428] Setting key 9ecc19d94ec348b6871358aace878068
Sep 25 02:53:26 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304806, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:26.238652 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:26.220739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:56025 (local address 127.2.172.126:46151)
0925 02:53:26.221291 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:26.221296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:26.222165 (+ 869us) server_negotiation.cc:407] Connection header received
0925 02:53:26.224481 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:26.224486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:26.224558 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:26.224701 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:26.226305 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.227112 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.228851 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.229333 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.229474 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:26.232996 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:26.233002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:26.233005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:26.233056 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:26.235527 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:26.236301 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:26.236305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:26.236307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:26.236429 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:26.236848 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:26.236851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:26.236853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:26.237104 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:26.237221 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:26.237910 (+ 689us) server_negotiation.cc:299] Negotiation successful
0925 02:53:26.238139 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":308,"threads_started":1}
I20260925 02:53:26.239858 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:26.221217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:46151 (local address 127.2.172.67:56025)
0925 02:53:26.221983 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:26.222032 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:26.224140 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:26.224956 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:26.224965 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:26.225398 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:26.226129 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:26.226147 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.227289 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.227293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:26.228669 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:26.228682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.229576 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.229583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:26.229873 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:26.230599 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:26.230622 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:26.232780 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:26.235711 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:26.235717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:26.235731 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:26.236189 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:26.236516 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:26.236520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:26.236522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:26.236751 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:26.237209 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:26.237215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:26.237519 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 02:53:26.237809 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 02:53:26.238350 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":325,"threads_started":1}
I20260925 02:53:26.241220 15808 heartbeater.cc:347] Connected to a master server at 127.2.172.126:46151
I20260925 02:53:26.241575 15808 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:26.242318 15808 heartbeater.cc:511] Master 127.2.172.126:46151 requested a full tablet report, sending...
I20260925 02:53:26.244067 15313 ts_manager.cc:194] Registered new tserver with Master: 1931d369be0d4d53bde9574bdeda1cbe (127.2.172.67:34809)
I20260925 02:53:26.244958 15313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:56025
I20260925 02:53:26.252470 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:53:26.254443 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 15394
Sep 25 02:53:26 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:53:26.273257 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:26.255486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.2.172.65:45769)
0925 02:53:26.256209 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:26.256217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:26.256246 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:53:26.256306 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:26.256314 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:26.256617 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:26.256760 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:26.258377 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.259467 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.261128 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.261707 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.261806 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:26.266798 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:26.266811 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:26.266826 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:26.266866 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:26.270362 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:26.271126 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:26.271134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:26.271136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:26.271312 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:26.271765 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:26.271769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:26.271771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:26.272132 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:26.272404 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:26.272590 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 02:53:26.272806 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":387,"threads_started":1}
I20260925 02:53:26.274631 15523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:26.358352 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 15394
I20260925 02:53:26.377046 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 15534
Sep 25 02:53:26 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:53:26.394977 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:26.378084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.2.172.66:39427)
0925 02:53:26.378780 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:26.378788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:26.378814 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:53:26.378877 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:26.378882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:26.379096 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:26.379229 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:26.380704 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.381773 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.383032 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.383663 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.383765 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:26.388852 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:26.388865 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:26.388879 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:26.388926 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:26.392010 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:26.392709 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:26.392717 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:26.392719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:26.392862 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:26.393263 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:26.393267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:26.393269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:26.393627 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:26.394002 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:26.394318 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:53:26.394596 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":421,"threads_started":1}
I20260925 02:53:26.396374 15667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:26.481837 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 15534
I20260925 02:53:26.502857 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 15678
Sep 25 02:53:26 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:53:26.519827 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:26.503894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.2.172.67:34809)
0925 02:53:26.504448 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:26.504456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:26.504483 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:53:26.504548 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:26.504554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:26.504766 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:26.504953 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:26.506262 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.507260 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.508653 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.509222 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.509364 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:26.514151 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:26.514163 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:26.514178 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:26.514214 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:26.517008 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:26.517735 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:26.517743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:26.517745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:26.517882 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:26.518229 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:26.518233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:26.518235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:26.518571 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:26.518914 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:26.519176 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 02:53:26.519381 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":258,"threads_started":1}
I20260925 02:53:26.521122 15803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:26.605079 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 15678
I20260925 02:53:26.621946 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 15280
I20260925 02:53:26.623306 15343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:26.729983 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 15280
I20260925 02:53:26.755988 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:46151
--webserver_interface=127.2.172.126
--webserver_port=46261
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:46151
--ranger_config_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.172.84:48717/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.172.84:48717/kms with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:27.084436 15820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:27.084743 15820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:27.084830 15820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:27.092271 15820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:27.092393 15820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:27.092439 15820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:27.092494 15820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:27.092541 15820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:27.102828 15820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.172.84:48717/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:46151
--ranger_config_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:46151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=46261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:27.104156 15820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:27.105418 15820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:27.117451 15825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:27.117486 15828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:27.117617 15826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:27.118311 15820 server_base.cc:1034] running on GCE node
I20260925 02:53:27.119035 15820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:27.120491 15820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:27.121730 15820 hybrid_clock.cc:648] HybridClock initialized: now 1790304807121680 us; error 48 us; skew 500 ppm
I20260925 02:53:27.122078 15820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:27 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304807, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:27.125344 15820 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:53:27.126974 15820 webserver.cc:460] Webserver started at http://127.2.172.126:46261/ using document root <none> and password file <none>
I20260925 02:53:27.127725 15820 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:27.127871 15820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:53:27 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304807, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:53:27.196458 15820 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:53:27.199692 15836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:27.201155 15820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:53:27.201352 15820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "38df122079a248b7b01be87495536928"
format_stamp: "Formatted at 2026-09-25 02:53:23 on dist-test-slave-g1xh"
server_key: "1d515be175c6c5498278a6210f3df7e7"
server_key_iv: "c6f9b2507d08961c9d66f0df04ea93cf"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:27.202294 15820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:27.214767 15820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:27.218689 15820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:27.218981 15820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:27.238144 15820 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:46151
I20260925 02:53:27.238188 15887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:46151 every 8 connection(s)
I20260925 02:53:27.239742 15820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:27.241524 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 15820
I20260925 02:53:27.242781 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.172.84:48717/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:45769
--local_ip_for_outbound_sockets=127.2.172.65
--tserver_master_addrs=127.2.172.126:46151
--webserver_port=43451
--webserver_interface=127.2.172.65
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.172.84:48717/kms with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:53:27.250798 15888 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:53:27.255187 15888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928: Bootstrap starting.
I20260925 02:53:27.276762 15888 log.cc:824] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:27.286623 15888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:53:27.287247 15888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928: Bootstrap complete.
I20260925 02:53:27.297928 15888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38df122079a248b7b01be87495536928" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46151 } }
I20260925 02:53:27.299047 15888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38df122079a248b7b01be87495536928, State: Initialized, Role: FOLLOWER
I20260925 02:53:27.299882 15888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38df122079a248b7b01be87495536928" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46151 } }
I20260925 02:53:27.300182 15888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:27.300338 15888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:27.300547 15888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:53:27.305516 15888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38df122079a248b7b01be87495536928" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46151 } }
I20260925 02:53:27.306172 15888 leader_election.cc:302] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38df122079a248b7b01be87495536928; no voters:
I20260925 02:53:27.306945 15888 leader_election.cc:288] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:53:27.307128 15892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:53:27.309113 15888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:27.313829 15892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [term 2 LEADER]: Becoming Leader. State: Replica: 38df122079a248b7b01be87495536928, State: Running, Role: LEADER
I20260925 02:53:27.314445 15892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38df122079a248b7b01be87495536928" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46151 } }
I20260925 02:53:27.325074 15893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "38df122079a248b7b01be87495536928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38df122079a248b7b01be87495536928" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46151 } } }
I20260925 02:53:27.324962 15894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38df122079a248b7b01be87495536928. Latest consensus state: current_term: 2 leader_uuid: "38df122079a248b7b01be87495536928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38df122079a248b7b01be87495536928" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 46151 } } }
I20260925 02:53:27.325501 15894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:27.325503 15893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:27.326565 15904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:27.338552 15904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:27.341063 15904 catalog_manager.cc:1259] Loaded cluster ID: 1c36a996eed946bf8ca570fd26038eff
I20260925 02:53:27.341320 15904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:27.347642 15904 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:27.350358 15904 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928: Loaded TSK: 0
I20260925 02:53:27.352702 15904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:53:27.355408 15888 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 02:53:27.706379 15890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:27.706940 15890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:27.707167 15890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:27.719671 15890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:27.719911 15890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:27.720168 15890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:53:27.736312 15890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.172.84:48717/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:45769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=43451
--tserver_master_addrs=127.2.172.126:46151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:27.738368 15890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:27.740334 15890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:27.769562 15926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:27.773125 15927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:27.780694 15890 server_base.cc:1034] running on GCE node
W20260925 02:53:27.783883 15929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:27.784964 15890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:27.792423 15890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:27.793756 15890 hybrid_clock.cc:648] HybridClock initialized: now 1790304807793685 us; error 83 us; skew 500 ppm
I20260925 02:53:27.794272 15890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:27 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304807, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:27.804064 15890 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:53:27.828663 15890 webserver.cc:460] Webserver started at http://127.2.172.65:43451/ using document root <none> and password file <none>
I20260925 02:53:27.829838 15890 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:27.830104 15890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:53:27 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304807, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:53:27.885033 15890 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.005s sys 0.005s
I20260925 02:53:27.895783 15938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:27.898010 15890 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.006s
I20260925 02:53:27.898355 15890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "f23c39320d4f4d66a22ae59bdcbe5700"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-g1xh"
server_key: "74d3f7042c3e552bdf311eeb4b027f70"
server_key_iv: "b4b3ecf6e9af16297c21493e2ade6d51"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:27.900074 15890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:27.942041 15890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:27.947890 15890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:27.948318 15890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:27.949766 15890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:27.952482 15890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:27.952616 15890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:27.952742 15890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:27.952876 15890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 02:53:28.003880 15890 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:45769
I20260925 02:53:28.003912 16052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:45769 every 8 connection(s)
I20260925 02:53:28.005820 15890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:28.007472 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 15890
I20260925 02:53:28.008332 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.172.84:48717/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:39427
--local_ip_for_outbound_sockets=127.2.172.66
--tserver_master_addrs=127.2.172.126:46151
--webserver_port=36283
--webserver_interface=127.2.172.66
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.172.84:48717/kms with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:53:28 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304807, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:28.055428 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:28.009630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:46151 (local address 127.2.172.65:39685)
0925 02:53:28.029348 (+ 19718us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:28.029404 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:28.031430 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:28.034338 (+ 2908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:28.034352 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:28.035025 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:28.035951 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:28.035997 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.039016 (+ 3019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.039023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:28.041744 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:28.041760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.042897 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.042904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:28.043184 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:28.043977 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:28.044002 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:28.046440 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:28.050117 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:28.050127 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:28.050144 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:28.050594 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:28.051084 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:28.051089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:28.051093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:28.051404 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:28.052590 (+ 1186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:28.052599 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:28.053020 (+ 421us) client_negotiation.cc:769] Sending connection context
0925 02:53:28.053353 (+ 333us) client_negotiation.cc:240] Negotiation successful
0925 02:53:28.053979 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18735,"thread_start_us":18495,"threads_started":1}
I20260925 02:53:28.055742 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:28.010850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:39685 (local address 127.2.172.126:46151)
0925 02:53:28.032554 (+ 21704us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:28.032576 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:28.032629 (+ 53us) server_negotiation.cc:407] Connection header received
0925 02:53:28.033061 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:28.033082 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:28.033627 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:28.036284 (+ 2657us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:28.036752 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.038804 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.041934 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.042617 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.042846 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:28.046798 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:28.046807 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:28.046823 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:28.046866 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:28.049935 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:28.050776 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:28.050783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:28.050790 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:28.050943 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:28.051588 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:28.051594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:28.051598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:28.052271 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:28.052488 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:28.053687 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0925 02:53:28.054102 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21385,"thread_start_us":22538,"threads_started":1}
I20260925 02:53:28.060274 16053 heartbeater.cc:347] Connected to a master server at 127.2.172.126:46151
I20260925 02:53:28.060631 16053 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:28.061616 16053 heartbeater.cc:511] Master 127.2.172.126:46151 requested a full tablet report, sending...
W20260925 02:53:28.065253 16053 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:46151 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:53:28.065505 16053 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:28.065815 16053 heartbeater.cc:511] Master 127.2.172.126:46151 requested a full tablet report, sending...
I20260925 02:53:28.067193 16053 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:28.067656 16053 heartbeater.cc:511] Master 127.2.172.126:46151 requested a full tablet report, sending...
W20260925 02:53:28.068825 16053 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:53:28.502302 16057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:28.502691 16057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:28.502902 16057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:28.512735 16057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:28.512897 16057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:28.513092 16057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:53:28.524001 16057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.172.84:48717/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:39427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=36283
--tserver_master_addrs=127.2.172.126:46151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:28.525456 16057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:28.526882 16057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:28.541955 16071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:28.541939 16072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:28.542884 16074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:28.543028 16057 server_base.cc:1034] running on GCE node
I20260925 02:53:28.543867 16057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:28.544987 16057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:28.546226 16057 hybrid_clock.cc:648] HybridClock initialized: now 1790304808546171 us; error 45 us; skew 500 ppm
I20260925 02:53:28.546627 16057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:28 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304808, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:28.550107 16057 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:53:28.551900 16057 webserver.cc:460] Webserver started at http://127.2.172.66:36283/ using document root <none> and password file <none>
I20260925 02:53:28.552743 16057 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:28.552904 16057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:53:28 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304808, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:53:28.598959 16057 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.000s sys 0.015s
I20260925 02:53:28.604187 16084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:28.608245 16057 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.001s
I20260925 02:53:28.608534 16057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "3337e40d25924f899b39b9ecee5d41ab"
format_stamp: "Formatted at 2026-09-25 02:53:25 on dist-test-slave-g1xh"
server_key: "402ab7c05832d30d081a7bf14b8e7f59"
server_key_iv: "bce5c2004eae655db4646c42575e9124"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:28.610170 16057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:28.654505 16057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:28.659246 16057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:28.659674 16057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:28.660863 16057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:28.662622 16057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:28.662744 16057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:28.662878 16057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:28.662978 16057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:53:28 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304808, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:28.738855 16057 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:39427
I20260925 02:53:28.738946 16196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:39427 every 8 connection(s)
I20260925 02:53:28.740904 16057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:28.751943 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 16057
I20260925 02:53:28.752875 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.172.84:48717/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:34809
--local_ip_for_outbound_sockets=127.2.172.67
--tserver_master_addrs=127.2.172.126:46151
--webserver_port=33209
--webserver_interface=127.2.172.67
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.172.84:48717/kms with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:53:28 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304808, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:28.799268 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:28.755801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:40915 (local address 127.2.172.126:46151)
0925 02:53:28.777959 (+ 22158us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:28.777970 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:28.782076 (+ 4106us) server_negotiation.cc:407] Connection header received
0925 02:53:28.784322 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:28.784328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:28.784416 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:28.784522 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:28.786261 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.786976 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.788648 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.789082 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.789227 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:28.793167 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:28.793173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:28.793180 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:28.793214 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:28.795965 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:28.796696 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:28.796701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:28.796703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:28.796822 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:28.797289 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:28.797294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:28.797298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:28.797603 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:28.797773 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:28.798550 (+ 777us) server_negotiation.cc:299] Negotiation successful
0925 02:53:28.798778 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21929,"thread_start_us":22172,"threads_started":1}
I20260925 02:53:28.800414 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:28.750891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:46151 (local address 127.2.172.66:40915)
0925 02:53:28.778895 (+ 28004us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:28.778972 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:28.783951 (+ 4979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:28.785032 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:28.785039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:28.785493 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:28.786072 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:28.786099 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.787140 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.787145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:28.788477 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:28.788498 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.789318 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.789328 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:28.789887 (+ 559us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:28.790642 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:28.790664 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:28.792902 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:28.796110 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:28.796119 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:28.796136 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:28.796561 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:28.796935 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:28.796940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:28.796944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:28.797167 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:28.797737 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:28.797745 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:28.798103 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 02:53:28.798372 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 02:53:28.798920 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27524,"thread_start_us":31104,"threads_started":1}
I20260925 02:53:28.802317 16197 heartbeater.cc:347] Connected to a master server at 127.2.172.126:46151
I20260925 02:53:28.802655 16197 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:28.803377 16197 heartbeater.cc:511] Master 127.2.172.126:46151 requested a full tablet report, sending...
I20260925 02:53:28.805410 15853 ts_manager.cc:194] Registered new tserver with Master: 3337e40d25924f899b39b9ecee5d41ab (127.2.172.66:39427)
I20260925 02:53:28.807333 15853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:40915
I20260925 02:53:29.068980 16053 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:29.069469 16053 heartbeater.cc:511] Master 127.2.172.126:46151 requested a full tablet report, sending...
I20260925 02:53:29.070883 15853 ts_manager.cc:194] Registered new tserver with Master: f23c39320d4f4d66a22ae59bdcbe5700 (127.2.172.65:45769)
I20260925 02:53:29.072134 15853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:39685
W20260925 02:53:29.211225 16201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:29.211661 16201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:29.211822 16201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 25 02:53:29 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304808, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
W20260925 02:53:29.223248 16201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:29.223503 16201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:29.223778 16201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:53:29.238883 16201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.172.84:48717/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:34809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=33209
--tserver_master_addrs=127.2.172.126:46151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:29.240608 16201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:29.242307 16201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:29.263207 16208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:29.269816 16207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:29.274996 16210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:29.275302 16201 server_base.cc:1034] running on GCE node
I20260925 02:53:29.276221 16201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:29.277678 16201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:29.279003 16201 hybrid_clock.cc:648] HybridClock initialized: now 1790304809278930 us; error 62 us; skew 500 ppm
I20260925 02:53:29.279475 16201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:29 dist-test-slave-g1xh krb5kdc[14830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304809, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:29.283877 16201 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:53:29.292879 16201 webserver.cc:460] Webserver started at http://127.2.172.67:33209/ using document root <none> and password file <none>
I20260925 02:53:29.293776 16201 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:29.293967 16201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:53:29 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304809, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:53:29.321573 16201 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.003s
I20260925 02:53:29.324681 16218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:29.326519 16201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:53:29.326704 16201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "1931d369be0d4d53bde9574bdeda1cbe"
format_stamp: "Formatted at 2026-09-25 02:53:26 on dist-test-slave-g1xh"
server_key: "b4e633f364e9629cad397280e4adaa42"
server_key_iv: "95d76ca13c66a617cbc2088d9427f95c"
server_key_version: "kuduclusterkey@0"
I20260925 02:53:29.328212 16201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:29.346627 16201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:29.350950 16201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:29.351254 16201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:29.352419 16201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:29.353901 16201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:29.353986 16201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:29.354149 16201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:29.354229 16201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:29.432443 16201 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:34809
I20260925 02:53:29.432662 16332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:34809 every 8 connection(s)
I20260925 02:53:29.434389 16201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:29.435410 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 16201
I20260925 02:53:29.470119 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.436450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.2.172.126:46151)
0925 02:53:29.450014 (+ 13564us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.450021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.450950 (+ 929us) server_negotiation.cc:407] Connection header received
0925 02:53:29.451032 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.451037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.451122 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.451227 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:29.453568 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.454700 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.456763 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.457437 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.457539 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.461231 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:29.461238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:29.461243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:29.461306 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:29.465594 (+ 4288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:29.466560 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:29.466567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:29.466571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:29.466720 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:29.467860 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:29.467874 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:29.467877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:29.468229 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:29.468491 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:29.469484 (+ 993us) server_negotiation.cc:299] Negotiation successful
0925 02:53:29.469748 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13402,"thread_start_us":13587,"threads_started":1}
Sep 25 02:53:29 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304809, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:29.492738 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.460055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:35829 (local address 127.2.172.126:46151)
0925 02:53:29.470455 (+ 10400us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.470460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.470476 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:53:29.470520 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.470525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.470616 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.470706 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:29.473938 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.474765 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.476647 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.477061 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.477198 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.481176 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:29.481184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:29.481188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:29.481232 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:29.487976 (+ 6744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:29.489643 (+ 1667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:29.489649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:29.489652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:29.489827 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:29.490435 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:29.490440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:29.490444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:29.490822 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:29.490976 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:29.492111 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0925 02:53:29.492377 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10272,"thread_start_us":11622,"threads_started":1}
I20260925 02:53:29.494230 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.459269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:46151 (local address 127.2.172.67:35829)
0925 02:53:29.464236 (+ 4967us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:29.464304 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:29.466808 (+ 2504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:29.472227 (+ 5419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:29.472240 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:29.472827 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:29.473728 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.473758 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.474934 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.474940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.476474 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.476487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.477301 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.477308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.477551 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.478276 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:29.478301 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:29.480835 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:29.488172 (+ 7337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:29.488183 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:29.488205 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:29.488871 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:29.489967 (+ 1096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:29.489973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:29.489976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:29.490283 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:29.491060 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:29.491081 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:29.491626 (+ 545us) client_negotiation.cc:769] Sending connection context
0925 02:53:29.491965 (+ 339us) client_negotiation.cc:240] Negotiation successful
0925 02:53:29.492595 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4581,"thread_start_us":4290,"threads_started":1}
I20260925 02:53:29.496369 16333 heartbeater.cc:347] Connected to a master server at 127.2.172.126:46151
I20260925 02:53:29.496716 16333 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:29.497653 16333 heartbeater.cc:511] Master 127.2.172.126:46151 requested a full tablet report, sending...
I20260925 02:53:29.499464 15853 ts_manager.cc:194] Registered new tserver with Master: 1931d369be0d4d53bde9574bdeda1cbe (127.2.172.67:34809)
I20260925 02:53:29.500629 15853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:35829
I20260925 02:53:29.504941 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:53:29.539954 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.510029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.2.172.126:46151)
0925 02:53:29.510249 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.510256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.515585 (+ 5329us) server_negotiation.cc:407] Connection header received
0925 02:53:29.515837 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.515843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.515927 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.516033 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:29.518291 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.519384 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.521664 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.522334 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.522537 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.526466 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:29.526473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:29.526478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:29.526526 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:29.535938 (+ 9412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:29.537148 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:29.537155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:29.537159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:29.537336 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:29.537849 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:29.537858 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:29.537863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:29.538304 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:29.538593 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:29.539225 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 02:53:29.539534 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
Sep 25 02:53:29 dist-test-slave-g1xh krb5kdc[14830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304808, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for HTTP/127.2.172.84@KRBTEST.COM
I20260925 02:53:29.547674 15853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:53:29.559129 15853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:29.617957 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.584545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45769 (local address 127.0.0.1:57374)
0925 02:53:29.591107 (+ 6562us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:29.591161 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:29.591358 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:29.592989 (+ 1631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:29.592997 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:29.593031 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:29.593457 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.593473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.605353 (+ 11880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.605358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.607566 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.607580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.616780 (+ 9200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.616787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.617044 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.617109 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 02:53:29.617274 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 02:53:29.617562 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6260,"thread_start_us":6463,"threads_started":1}
I20260925 02:53:29.618778 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.585198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.2.172.65:45769)
0925 02:53:29.585935 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.585948 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.591619 (+ 5671us) server_negotiation.cc:407] Connection header received
0925 02:53:29.591728 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.591736 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.592009 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.592206 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:29.594349 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.605159 (+ 10810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.607766 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.616564 (+ 8798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.616725 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.616997 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:29.618229 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0925 02:53:29.618435 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":765,"threads_started":1}
I20260925 02:53:29.623272 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.584975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39427 (local address 127.0.0.1:36052)
0925 02:53:29.594043 (+ 9068us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:29.594065 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:29.594174 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:29.597276 (+ 3102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:29.597281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:29.597307 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:29.597697 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.597711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.602365 (+ 4654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.602371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.604837 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.604857 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.622445 (+ 17588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.622453 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.622635 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.622669 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:53:29.622773 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:53:29.622920 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8908,"thread_start_us":8775,"threads_started":1}
I20260925 02:53:29.624164 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.585641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.2.172.66:39427)
0925 02:53:29.596385 (+ 10744us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.596395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.596441 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:53:29.596539 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.596547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.596803 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.597007 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:29.600646 (+ 3639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.602184 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.615660 (+ 13476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.620270 (+ 4610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.620477 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.620751 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:29.623660 (+ 2909us) server_negotiation.cc:299] Negotiation successful
0925 02:53:29.623852 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10567,"thread_start_us":10441,"threads_started":1}
I20260925 02:53:29.624337 15987 tablet_service.cc:1467] Processing CreateTablet for tablet a726ba9661a841a5ba4946ee1a89524a (DEFAULT_TABLE table=test-table [id=e467c66822544cb98699ad619ae33ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:29.629943 15987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726ba9661a841a5ba4946ee1a89524a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:29.631897 16132 tablet_service.cc:1467] Processing CreateTablet for tablet a726ba9661a841a5ba4946ee1a89524a (DEFAULT_TABLE table=test-table [id=e467c66822544cb98699ad619ae33ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:29.633437 16132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726ba9661a841a5ba4946ee1a89524a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:29.655732 16355 tablet_bootstrap.cc:492] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab: Bootstrap starting.
I20260925 02:53:29.656178 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.594482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.2.172.67:34809)
0925 02:53:29.621297 (+ 26815us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.621309 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.621352 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:53:29.621486 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.621498 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.621820 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.622076 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:29.637075 (+ 14999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.638877 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.651756 (+ 12879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.653987 (+ 2231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.654174 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.654525 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:29.655390 (+ 865us) server_negotiation.cc:299] Negotiation successful
0925 02:53:29.655657 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26613,"thread_start_us":26463,"threads_started":1}
I20260925 02:53:29.657238 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.596727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:34809 (local address 127.0.0.1:49608)
0925 02:53:29.619532 (+ 22805us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:29.619590 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:29.619702 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:29.636596 (+ 16894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:29.636602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:29.636638 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:29.636828 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.636839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.639104 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.639109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.641065 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.641083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.654769 (+ 13686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.654779 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.655093 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.655150 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 02:53:29.656548 (+ 1398us) client_negotiation.cc:240] Negotiation successful
0925 02:53:29.656808 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22693,"thread_start_us":30419,"threads_started":1}
I20260925 02:53:29.662389 16267 tablet_service.cc:1467] Processing CreateTablet for tablet a726ba9661a841a5ba4946ee1a89524a (DEFAULT_TABLE table=test-table [id=e467c66822544cb98699ad619ae33ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:29.664060 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726ba9661a841a5ba4946ee1a89524a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:29.664942 16356 tablet_bootstrap.cc:492] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700: Bootstrap starting.
I20260925 02:53:29.669351 16356 tablet_bootstrap.cc:654] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:29.670748 16356 log.cc:824] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:29.671269 16355 tablet_bootstrap.cc:654] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:29.672931 16355 log.cc:824] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:29.681226 16356 tablet_bootstrap.cc:492] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700: No bootstrap required, opened a new log
I20260925 02:53:29.681734 16356 ts_tablet_manager.cc:1397] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260925 02:53:29.686174 16359 tablet_bootstrap.cc:492] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe: Bootstrap starting.
I20260925 02:53:29.689652 16355 tablet_bootstrap.cc:492] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab: No bootstrap required, opened a new log
I20260925 02:53:29.689960 16355 ts_tablet_manager.cc:1397] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.009s
I20260925 02:53:29.692231 16359 tablet_bootstrap.cc:654] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:29.693688 16359 log.cc:824] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:29.697803 16356 raft_consensus.cc:348] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45769 } } peers { permanent_uuid: "3337e40d25924f899b39b9ecee5d41ab" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39427 } } peers { permanent_uuid: "1931d369be0d4d53bde9574bdeda1cbe" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 34809 } }
I20260925 02:53:29.698460 16356 raft_consensus.cc:374] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:29.698624 16356 raft_consensus.cc:729] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23c39320d4f4d66a22ae59bdcbe5700, State: Initialized, Role: FOLLOWER
I20260925 02:53:29.699478 16356 consensus_queue.cc:260] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45769 } } peers { permanent_uuid: "3337e40d25924f899b39b9ecee5d41ab" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39427 } } peers { permanent_uuid: "1931d369be0d4d53bde9574bdeda1cbe" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 34809 } }
I20260925 02:53:29.703497 16359 tablet_bootstrap.cc:492] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe: No bootstrap required, opened a new log
I20260925 02:53:29.703967 16359 ts_tablet_manager.cc:1397] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.008s
I20260925 02:53:29.704310 16355 raft_consensus.cc:348] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45769 } } peers { permanent_uuid: "3337e40d25924f899b39b9ecee5d41ab" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39427 } } peers { permanent_uuid: "1931d369be0d4d53bde9574bdeda1cbe" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 34809 } }
I20260925 02:53:29.704945 16355 raft_consensus.cc:374] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:29.705116 16355 raft_consensus.cc:729] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3337e40d25924f899b39b9ecee5d41ab, State: Initialized, Role: FOLLOWER
I20260925 02:53:29.706360 16355 consensus_queue.cc:260] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45769 } } peers { permanent_uuid: "3337e40d25924f899b39b9ecee5d41ab" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39427 } } peers { permanent_uuid: "1931d369be0d4d53bde9574bdeda1cbe" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 34809 } }
I20260925 02:53:29.714365 16197 heartbeater.cc:503] Master 127.2.172.126:46151 was elected leader, sending a full tablet report...
I20260925 02:53:29.714713 16355 ts_tablet_manager.cc:1428] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab: Time spent starting tablet: real 0.025s user 0.010s sys 0.012s
I20260925 02:53:29.715853 16053 heartbeater.cc:503] Master 127.2.172.126:46151 was elected leader, sending a full tablet report...
I20260925 02:53:29.716629 16356 ts_tablet_manager.cc:1428] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700: Time spent starting tablet: real 0.035s user 0.028s sys 0.000s
I20260925 02:53:29.718499 16359 raft_consensus.cc:348] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45769 } } peers { permanent_uuid: "3337e40d25924f899b39b9ecee5d41ab" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39427 } } peers { permanent_uuid: "1931d369be0d4d53bde9574bdeda1cbe" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 34809 } }
I20260925 02:53:29.719092 16359 raft_consensus.cc:374] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:29.719237 16359 raft_consensus.cc:729] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1931d369be0d4d53bde9574bdeda1cbe, State: Initialized, Role: FOLLOWER
I20260925 02:53:29.721117 16359 consensus_queue.cc:260] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45769 } } peers { permanent_uuid: "3337e40d25924f899b39b9ecee5d41ab" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39427 } } peers { permanent_uuid: "1931d369be0d4d53bde9574bdeda1cbe" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 34809 } }
I20260925 02:53:29.722693 16333 heartbeater.cc:503] Master 127.2.172.126:46151 was elected leader, sending a full tablet report...
I20260925 02:53:29.723035 16359 ts_tablet_manager.cc:1428] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe: Time spent starting tablet: real 0.019s user 0.012s sys 0.001s
W20260925 02:53:29.777926 16054 tablet.cc:2369] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:53:29.785137 16198 tablet.cc:2369] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:29.803056 16362 raft_consensus.cc:482] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:29.803406 16362 raft_consensus.cc:504] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45769 } } peers { permanent_uuid: "3337e40d25924f899b39b9ecee5d41ab" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39427 } } peers { permanent_uuid: "1931d369be0d4d53bde9574bdeda1cbe" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 34809 } }
I20260925 02:53:29.804286 16361 raft_consensus.cc:482] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:29.804553 16361 raft_consensus.cc:504] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45769 } } peers { permanent_uuid: "3337e40d25924f899b39b9ecee5d41ab" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39427 } } peers { permanent_uuid: "1931d369be0d4d53bde9574bdeda1cbe" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 34809 } }
I20260925 02:53:29.805773 16362 leader_election.cc:288] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23c39320d4f4d66a22ae59bdcbe5700 (127.2.172.65:45769), 1931d369be0d4d53bde9574bdeda1cbe (127.2.172.67:34809)
I20260925 02:53:29.806324 16361 leader_election.cc:288] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3337e40d25924f899b39b9ecee5d41ab (127.2.172.66:39427), 1931d369be0d4d53bde9574bdeda1cbe (127.2.172.67:34809)
I20260925 02:53:29.853205 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.807675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:52305 (local address 127.2.172.65:45769)
0925 02:53:29.807849 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.807856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.830493 (+ 22637us) server_negotiation.cc:407] Connection header received
0925 02:53:29.831681 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.831688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.831806 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.831897 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:29.838128 (+ 6231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.839291 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.847190 (+ 7899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.848791 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.848964 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.849091 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:29.852659 (+ 3568us) server_negotiation.cc:299] Negotiation successful
0925 02:53:29.852876 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:53:29.854386 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.806570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45769 (local address 127.2.172.66:52305)
0925 02:53:29.830333 (+ 23763us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:29.830363 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:29.831459 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:29.832177 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:29.832182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:29.832217 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:29.832524 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.837617 (+ 5093us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.839938 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.839943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.844924 (+ 4981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.844942 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.849306 (+ 4364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.849314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.849550 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.849603 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 02:53:29.853721 (+ 4118us) client_negotiation.cc:240] Negotiation successful
0925 02:53:29.853928 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23591,"thread_start_us":23940,"threads_started":1}
I20260925 02:53:29.855684 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.808601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:49825 (local address 127.2.172.67:34809)
0925 02:53:29.808790 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.808796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.831327 (+ 22531us) server_negotiation.cc:407] Connection header received
0925 02:53:29.836436 (+ 5109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.836442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.836525 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.836595 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:29.841237 (+ 4642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.842263 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.849786 (+ 7523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.851383 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.851542 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.851682 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:29.855171 (+ 3489us) server_negotiation.cc:299] Negotiation successful
0925 02:53:29.855377 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:53:29.858371 16007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726ba9661a841a5ba4946ee1a89524a" candidate_uuid: "3337e40d25924f899b39b9ecee5d41ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" is_pre_election: true
I20260925 02:53:29.858949 16007 raft_consensus.cc:2454] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3337e40d25924f899b39b9ecee5d41ab in term 0.
I20260925 02:53:29.859964 16085 leader_election.cc:302] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3337e40d25924f899b39b9ecee5d41ab, f23c39320d4f4d66a22ae59bdcbe5700; no voters:
I20260925 02:53:29.860538 16362 raft_consensus.cc:2781] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:29.860695 16362 raft_consensus.cc:482] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:29.860793 16362 raft_consensus.cc:3037] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:29.864588 16362 raft_consensus.cc:504] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45769 } } peers { permanent_uuid: "3337e40d25924f899b39b9ecee5d41ab" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39427 } } peers { permanent_uuid: "1931d369be0d4d53bde9574bdeda1cbe" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 34809 } }
I20260925 02:53:29.865412 16362 leader_election.cc:288] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [CANDIDATE]: Term 1 election: Requested vote from peers f23c39320d4f4d66a22ae59bdcbe5700 (127.2.172.65:45769), 1931d369be0d4d53bde9574bdeda1cbe (127.2.172.67:34809)
I20260925 02:53:29.866392 16007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726ba9661a841a5ba4946ee1a89524a" candidate_uuid: "3337e40d25924f899b39b9ecee5d41ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23c39320d4f4d66a22ae59bdcbe5700"
I20260925 02:53:29.866619 16007 raft_consensus.cc:3037] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:29.870810 16007 raft_consensus.cc:2454] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3337e40d25924f899b39b9ecee5d41ab in term 1.
I20260925 02:53:29.871507 16085 leader_election.cc:302] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3337e40d25924f899b39b9ecee5d41ab, f23c39320d4f4d66a22ae59bdcbe5700; no voters:
I20260925 02:53:29.871965 16362 raft_consensus.cc:2781] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:29.857338 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.808149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:34809 (local address 127.2.172.66:49825)
0925 02:53:29.831219 (+ 23070us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:29.831236 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:29.836233 (+ 4997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:29.836878 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:29.836883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:29.836910 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:29.839700 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.839716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.842444 (+ 2728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.842449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.846216 (+ 3767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.846231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.851887 (+ 5656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.851896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.852159 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.852212 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 02:53:29.856776 (+ 4564us) client_negotiation.cc:240] Negotiation successful
0925 02:53:29.856928 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22958,"thread_start_us":36926,"threads_started":1}
I20260925 02:53:29.874935 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726ba9661a841a5ba4946ee1a89524a" candidate_uuid: "3337e40d25924f899b39b9ecee5d41ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1931d369be0d4d53bde9574bdeda1cbe" is_pre_election: true
I20260925 02:53:29.875720 16287 raft_consensus.cc:2454] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3337e40d25924f899b39b9ecee5d41ab in term 0.
I20260925 02:53:29.876276 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.847346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:43657 (local address 127.2.172.67:34809)
0925 02:53:29.855592 (+ 8246us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.855601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.868214 (+ 12613us) server_negotiation.cc:407] Connection header received
0925 02:53:29.868529 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.868535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.868621 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.868698 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:29.869655 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.870797 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.873393 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.874797 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.874971 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.875082 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:29.875746 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 02:53:29.875892 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8012,"thread_start_us":7783,"threads_started":1}
I20260925 02:53:29.876896 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726ba9661a841a5ba4946ee1a89524a" candidate_uuid: "3337e40d25924f899b39b9ecee5d41ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1931d369be0d4d53bde9574bdeda1cbe"
I20260925 02:53:29.877100 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.853419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:34809 (local address 127.2.172.65:43657)
0925 02:53:29.868074 (+ 14655us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:29.868096 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:29.868339 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:29.869006 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:29.869010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:29.869038 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:29.869439 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.869451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.871012 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.871019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.873206 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.873223 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.875305 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.875313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.875539 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.875591 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 02:53:29.876597 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0925 02:53:29.876744 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14466,"thread_start_us":17992,"threads_started":1}
I20260925 02:53:29.877151 16287 raft_consensus.cc:3037] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:29.877751 16286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726ba9661a841a5ba4946ee1a89524a" candidate_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1931d369be0d4d53bde9574bdeda1cbe" is_pre_election: true
I20260925 02:53:29.881696 16287 raft_consensus.cc:2454] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3337e40d25924f899b39b9ecee5d41ab in term 1.
I20260925 02:53:29.892899 16362 raft_consensus.cc:686] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 1 LEADER]: Becoming Leader. State: Replica: 3337e40d25924f899b39b9ecee5d41ab, State: Running, Role: LEADER
I20260925 02:53:29.893709 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.807373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:38701 (local address 127.2.172.66:39427)
0925 02:53:29.807851 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.807857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.847062 (+ 39205us) server_negotiation.cc:407] Connection header received
0925 02:53:29.854742 (+ 7680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.854750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.854862 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.854932 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:29.882472 (+ 27540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.883930 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.890047 (+ 6117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.891763 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.891933 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.892046 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:29.893297 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0925 02:53:29.893441 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:53:29.894598 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.806829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39427 (local address 127.2.172.65:38701)
0925 02:53:29.846942 (+ 40113us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:29.846968 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:29.852463 (+ 5495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:29.856142 (+ 3679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:29.856148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:29.856181 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:29.856635 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.856651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.884367 (+ 27716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.884375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.886933 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.886958 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.892318 (+ 5360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.892326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.892553 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.892605 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 02:53:29.894087 (+ 1482us) client_negotiation.cc:240] Negotiation successful
0925 02:53:29.894253 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39940,"thread_start_us":39739,"threads_started":1}
I20260925 02:53:29.895853 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726ba9661a841a5ba4946ee1a89524a" candidate_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3337e40d25924f899b39b9ecee5d41ab" is_pre_election: true
I20260925 02:53:29.895885 16362 consensus_queue.cc:237] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45769 } } peers { permanent_uuid: "3337e40d25924f899b39b9ecee5d41ab" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39427 } } peers { permanent_uuid: "1931d369be0d4d53bde9574bdeda1cbe" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 34809 } }
I20260925 02:53:29.897123 15942 leader_election.cc:302] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f23c39320d4f4d66a22ae59bdcbe5700; no voters: 1931d369be0d4d53bde9574bdeda1cbe, 3337e40d25924f899b39b9ecee5d41ab
I20260925 02:53:29.897948 16361 raft_consensus.cc:2726] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 02:53:29.912204 15852 catalog_manager.cc:5495] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 3337e40d25924f899b39b9ecee5d41ab (127.2.172.66). New cstate: current_term: 1 leader_uuid: "3337e40d25924f899b39b9ecee5d41ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3337e40d25924f899b39b9ecee5d41ab" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1931d369be0d4d53bde9574bdeda1cbe" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 34809 } health_report { overall_health: UNKNOWN } } }
W20260925 02:53:29.969273 16336 tablet.cc:2369] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:29.983397 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.974694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.2.172.66:39427)
0925 02:53:29.974867 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.974873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.975489 (+ 616us) server_negotiation.cc:407] Connection header received
0925 02:53:29.975636 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.975641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.975751 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.975891 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:29.976844 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.977976 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.980106 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.980951 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.981128 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.981764 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:29.982602 (+ 838us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:29.982796 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:29.982859 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 02:53:29.982972 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:53:29.995489 16287 raft_consensus.cc:1261] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe [term 1 FOLLOWER]: Refusing update from remote peer 3337e40d25924f899b39b9ecee5d41ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:29.995908 16007 raft_consensus.cc:1261] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [term 1 FOLLOWER]: Refusing update from remote peer 3337e40d25924f899b39b9ecee5d41ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:29.996654 16362 consensus_queue.cc:1035] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1931d369be0d4d53bde9574bdeda1cbe" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:29.997071 16369 consensus_queue.cc:1035] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23c39320d4f4d66a22ae59bdcbe5700" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:30.037117 16380 mvcc.cc:205] Tried to move back new op lower bound from 7333088501730086912 to 7333088501338857472. Current Snapshot: MvccSnapshot[applied={T|T < 7333088501730086912}]
I20260925 02:53:30.060132 16381 mvcc.cc:205] Tried to move back new op lower bound from 7333088501730086912 to 7333088501338857472. Current Snapshot: MvccSnapshot[applied={T|T < 7333088501730086912}]
I20260925 02:53:30.073172 15852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:30.073546 15852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:30.079371 15852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38df122079a248b7b01be87495536928: Sending DeleteTablet for 3 replicas of tablet a726ba9661a841a5ba4946ee1a89524a
I20260925 02:53:30.080994 15987 tablet_service.cc:1514] Processing DeleteTablet for tablet a726ba9661a841a5ba4946ee1a89524a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:30 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:57374
I20260925 02:53:30.085065 16132 tablet_service.cc:1514] Processing DeleteTablet for tablet a726ba9661a841a5ba4946ee1a89524a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:30 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:36052
I20260925 02:53:30.088171 16389 tablet_replica.cc:330] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700: stopping tablet replica
I20260925 02:53:30.088771 16389 raft_consensus.cc:2229] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:30.089418 16389 raft_consensus.cc:2258] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:30.091368 16267 tablet_service.cc:1514] Processing DeleteTablet for tablet a726ba9661a841a5ba4946ee1a89524a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:30 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:49608
I20260925 02:53:30.094245 16390 tablet_replica.cc:330] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab: stopping tablet replica
I20260925 02:53:30.094799 16390 raft_consensus.cc:2229] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:53:30.095642 16390 raft_consensus.cc:2258] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:30.099673 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 15890
I20260925 02:53:30.104298 16391 tablet_replica.cc:330] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe: stopping tablet replica
I20260925 02:53:30.104807 16391 raft_consensus.cc:2229] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:30.105365 16391 raft_consensus.cc:2258] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:30.108199 16391 ts_tablet_manager.cc:1905] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:30.109915 16389 ts_tablet_manager.cc:1905] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:30.110075 16390 ts_tablet_manager.cc:1905] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:30.125083 16390 ts_tablet_manager.cc:1918] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:30.125811 16389 ts_tablet_manager.cc:1918] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:30.126094 16389 log.cc:1196] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/a726ba9661a841a5ba4946ee1a89524a
I20260925 02:53:30.127218 16390 log.cc:1196] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/a726ba9661a841a5ba4946ee1a89524a
I20260925 02:53:30.128180 16390 ts_tablet_manager.cc:1939] T a726ba9661a841a5ba4946ee1a89524a P 3337e40d25924f899b39b9ecee5d41ab: Deleting consensus metadata
I20260925 02:53:30.128716 16391 ts_tablet_manager.cc:1918] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:30.126900 16389 ts_tablet_manager.cc:1939] T a726ba9661a841a5ba4946ee1a89524a P f23c39320d4f4d66a22ae59bdcbe5700: Deleting consensus metadata
I20260925 02:53:30.133002 15840 catalog_manager.cc:4841] TS 3337e40d25924f899b39b9ecee5d41ab (127.2.172.66:39427): tablet a726ba9661a841a5ba4946ee1a89524a (table test-table [id=e467c66822544cb98699ad619ae33ac2]) successfully deleted
I20260925 02:53:30.133831 15837 catalog_manager.cc:4841] TS f23c39320d4f4d66a22ae59bdcbe5700 (127.2.172.65:45769): tablet a726ba9661a841a5ba4946ee1a89524a (table test-table [id=e467c66822544cb98699ad619ae33ac2]) successfully deleted
I20260925 02:53:30.138123 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:30.100964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.2.172.65:45769)
0925 02:53:30.101211 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:30.101221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:30.106463 (+ 5242us) server_negotiation.cc:407] Connection header received
0925 02:53:30.106693 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:30.106699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:30.106786 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:30.106889 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:30.113560 (+ 6671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.114791 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.116878 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.117789 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.117938 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:30.123353 (+ 5415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:30.123386 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:30.123406 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:30.123476 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:30.129182 (+ 5706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:30.135241 (+ 6059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:30.135253 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:30.135257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:30.135446 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:30.135958 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:30.135963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:30.135967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:30.136494 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:30.136879 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:30.137485 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 02:53:30.137734 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 02:53:30.139420 16391 log.cc:1196] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/a726ba9661a841a5ba4946ee1a89524a
I20260925 02:53:30.139904 16048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:30.140506 16391 ts_tablet_manager.cc:1939] T a726ba9661a841a5ba4946ee1a89524a P 1931d369be0d4d53bde9574bdeda1cbe: Deleting consensus metadata
I20260925 02:53:30.142445 15840 catalog_manager.cc:4841] TS 1931d369be0d4d53bde9574bdeda1cbe (127.2.172.67:34809): tablet a726ba9661a841a5ba4946ee1a89524a (table test-table [id=e467c66822544cb98699ad619ae33ac2]) successfully deleted
I20260925 02:53:30.324910 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 15890
I20260925 02:53:30.353277 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 16057
I20260925 02:53:30.373829 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:30.354161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.2.172.66:39427)
0925 02:53:30.354355 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:30.354362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:30.354440 (+ 78us) server_negotiation.cc:407] Connection header received
0925 02:53:30.354638 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:30.354644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:30.354754 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:30.354859 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:30.357100 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.358240 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.360239 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.360896 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.361036 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:30.364725 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:30.364741 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:30.364760 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:30.364810 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:30.369114 (+ 4304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:30.370433 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:30.370444 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:30.370448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:30.370633 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:30.371105 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:30.371110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:30.371114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:30.371645 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:30.371971 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:30.373193 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0925 02:53:30.373434 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:53:30.374984 16192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:30.523819 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 16057
I20260925 02:53:30.543246 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 16201
I20260925 02:53:30.558461 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:30.544257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.2.172.67:34809)
0925 02:53:30.544737 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:30.544742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:30.544760 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:53:30.544818 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:30.544822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:30.544903 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:30.545005 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:30.546326 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.547133 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.548621 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.549048 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.549156 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:30.551860 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:30.551877 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:30.551893 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:30.551931 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:30.555469 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:30.556278 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:30.556289 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:30.556292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:30.556450 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:30.556847 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:30.556854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:30.556857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:30.557229 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:30.557538 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:30.557859 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 02:53:30.558106 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":243,"threads_started":1}
I20260925 02:53:30.559175 16328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:30.649065 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 16201
I20260925 02:53:30.667011 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 15820
I20260925 02:53:30.668247 15883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:30.778322 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 15820
I20260925 02:53:30.802076 2737 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:53:31.138029 2737 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:53:31.138319 2737 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:53:31.484206 2737 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:53:31.484 UTC [14860] LOG: received smart shutdown request
2026-09-25 02:53:31.488 UTC [14860] LOG: background worker "logical replication launcher" (PID 14869) exited with exit code 1
2026-09-25 02:53:31.488 UTC [14864] LOG: shutting down
2026-09-25 02:53:31.672 UTC [14860] LOG: database system is shut down
2026-09-25T02:53:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54652 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:53:31 dist-test-slave-g1xh krb5kdc[16406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:53:31 dist-test-slave-g1xh krb5kdc[16406](info): set up 2 sockets
Sep 25 02:53:31 dist-test-slave-g1xh krb5kdc[16406](info): commencing operation
krb5kdc: starting...
W20260925 02:53:32.930172 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:53:33 dist-test-slave-g1xh krb5kdc[16406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:53:33.191105 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:41901
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:41901,127.2.172.125:45017,127.2.172.124:38529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:33.528667 16419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:33.528988 16419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:33.529191 16419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:33.536825 16419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:33.536947 16419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:33.537055 16419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:33.537160 16419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:33.537243 16419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:33.547870 16419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:41901,127.2.172.125:45017,127.2.172.124:38529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:41901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:33.549477 16419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:33.551762 16419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:33.565881 16424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:33.565946 16425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:33.565970 16427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:33.567095 16419 server_base.cc:1034] running on GCE node
I20260925 02:53:33.568130 16419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:33.569734 16419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:33.570947 16419 hybrid_clock.cc:648] HybridClock initialized: now 1790304813570906 us; error 34 us; skew 500 ppm
I20260925 02:53:33.571331 16419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:33 dist-test-slave-g1xh krb5kdc[16406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304813, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:33.581364 16419 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:53:33.583168 16419 webserver.cc:460] Webserver started at http://127.2.172.126:37593/ using document root <none> and password file <none>
I20260925 02:53:33.583976 16419 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:33.584088 16419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:33.584362 16419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:33.587468 16419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "44b73825c4a2425bb1a60b9002d4dbba"
format_stamp: "Formatted at 2026-09-25 02:53:33 on dist-test-slave-g1xh"
server_key: "4e25471fe9b23e4e12bd9f43c2113afb"
server_key_iv: "71a3001186d427daab353db8b243e180"
server_key_version: "encryptionkey@0"
I20260925 02:53:33.588392 16419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "44b73825c4a2425bb1a60b9002d4dbba"
format_stamp: "Formatted at 2026-09-25 02:53:33 on dist-test-slave-g1xh"
server_key: "4e25471fe9b23e4e12bd9f43c2113afb"
server_key_iv: "71a3001186d427daab353db8b243e180"
server_key_version: "encryptionkey@0"
I20260925 02:53:33.594103 16419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:53:33.598234 16434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:33.599990 16419 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:53:33.600152 16419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "44b73825c4a2425bb1a60b9002d4dbba"
format_stamp: "Formatted at 2026-09-25 02:53:33 on dist-test-slave-g1xh"
server_key: "4e25471fe9b23e4e12bd9f43c2113afb"
server_key_iv: "71a3001186d427daab353db8b243e180"
server_key_version: "encryptionkey@0"
I20260925 02:53:33.600339 16419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:33.621768 16419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:33.626600 16419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:33.626859 16419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:33.646770 16419 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:41901
I20260925 02:53:33.646867 16485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:41901 every 8 connection(s)
I20260925 02:53:33.648286 16419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:33.652263 16486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:33.657554 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 16419
I20260925 02:53:33.657800 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:53:33.658476 2737 external_mini_cluster.cc:1428] Setting key 640f6d35c39814643897b569e83b10d1
I20260925 02:53:33.659437 16486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:53:33 dist-test-slave-g1xh krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304813, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.172.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.125@KRBTEST.COM" created.
I20260925 02:53:33.686100 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.662905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:41901 (local address 127.0.0.1:36574)
0925 02:53:33.663870 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:33.663942 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:33.666731 (+ 2789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:33.667888 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:33.667894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:33.668481 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:33.668843 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:33.668861 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.671255 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.671266 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:33.672512 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:33.672526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.673989 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.673996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:33.674280 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:33.675048 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:33.675072 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:33.677714 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:33.681287 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:33.681303 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:33.681319 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:33.682015 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:33.682617 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:33.682625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:33.682631 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:33.682862 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:33.683836 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:33.683851 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:33.684065 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 02:53:33.684304 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 02:53:33.684874 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":182,"thread_start_us":374,"threads_started":1}
I20260925 02:53:33.686136 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.664161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.2.172.126:41901)
0925 02:53:33.664853 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:33.664859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:33.664900 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:53:33.667171 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:33.667181 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:33.667531 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:33.667835 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:33.669049 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.671050 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.672671 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.673710 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.673910 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:33.677980 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:33.677995 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:33.678018 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:33.678066 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:33.681131 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:33.682180 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:33.682212 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:33.682218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:33.682452 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:33.682989 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:33.682993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:33.682995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:33.683503 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:33.683891 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:33.684452 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 02:53:33.684733 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":527,"thread_start_us":376,"threads_started":1}
I20260925 02:53:33.689273 16486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:53:33.690536 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.690103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51216)
0925 02:53:33.690303 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:33.690412 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 02:53:33.690856 16436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
I20260925 02:53:33.692606 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.692223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51232)
0925 02:53:33.692390 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:33.692459 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 02:53:33.692819 16486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:45017: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.125.keytab.
Entry for principal kudu/127.2.172.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.125.keytab.
I20260925 02:53:33.718453 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.125:45017
--webserver_interface=127.2.172.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.125.keytab
--principal=kudu/127.2.172.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:41901,127.2.172.125:45017,127.2.172.124:38529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:53:33.742182 16486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } attempt: 1
I20260925 02:53:33.743458 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.742904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51242)
0925 02:53:33.743141 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:33.743282 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 02:53:33.744527 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.744112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51248)
0925 02:53:33.744257 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:33.744342 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 02:53:33.744715 16486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:45017: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
I20260925 02:53:33.812988 16486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } attempt: 2
I20260925 02:53:33.814327 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.813789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51262)
0925 02:53:33.814032 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:33.814146 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 02:53:33.815464 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.815049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51266)
0925 02:53:33.815218 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:33.815306 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 02:53:33.815742 16486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:45017: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
I20260925 02:53:33.907032 16486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } attempt: 3
I20260925 02:53:33.908404 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.907871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51276)
0925 02:53:33.908106 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:33.908233 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 02:53:33.909677 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.909208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51282)
0925 02:53:33.909404 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:33.909504 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 02:53:33.909976 16486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:45017: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
I20260925 02:53:34.053265 16486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } attempt: 4
I20260925 02:53:34.054527 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.053904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51294)
0925 02:53:34.054249 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.054357 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260925 02:53:34.055697 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.055250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51308)
0925 02:53:34.055405 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.055511 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 02:53:34.055915 16486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:45017: Network error: Client connection negotiation failed: client connection to 127.2.172.125:45017: connect: Connection refused (error 111)
W20260925 02:53:34.057375 16493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:34.057674 16493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:34.057838 16493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:34.065320 16493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:34.065423 16493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:34.065502 16493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:34.065610 16493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:34.065690 16493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:34.076010 16493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:41901,127.2.172.125:45017,127.2.172.124:38529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.125.keytab
--principal=kudu/127.2.172.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.125:45017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.172.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:34.077364 16493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:34.078660 16493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:34.090487 16498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:34.090646 16499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:34.090646 16501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:34.092052 16493 server_base.cc:1034] running on GCE node
I20260925 02:53:34.092893 16493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:34.094414 16493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:34.095669 16493 hybrid_clock.cc:648] HybridClock initialized: now 1790304814095624 us; error 53 us; skew 500 ppm
I20260925 02:53:34.096030 16493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304814, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:34.099359 16493 init.cc:377] Logged in from keytab as kudu/127.2.172.125@KRBTEST.COM (short username kudu)
I20260925 02:53:34.101029 16493 webserver.cc:460] Webserver started at http://127.2.172.125:44609/ using document root <none> and password file <none>
I20260925 02:53:34.101890 16493 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:34.102056 16493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:34.102377 16493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:34.105715 16493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/data/instance:
uuid: "d0dffd86c761454b8564bb1a15f24f4c"
format_stamp: "Formatted at 2026-09-25 02:53:34 on dist-test-slave-g1xh"
server_key: "00d978b638e6deae82f1b1f3b96879b3"
server_key_iv: "81e16123dc53dd04094eab7b30277962"
server_key_version: "encryptionkey@0"
I20260925 02:53:34.106549 16493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/wal/instance:
uuid: "d0dffd86c761454b8564bb1a15f24f4c"
format_stamp: "Formatted at 2026-09-25 02:53:34 on dist-test-slave-g1xh"
server_key: "00d978b638e6deae82f1b1f3b96879b3"
server_key_iv: "81e16123dc53dd04094eab7b30277962"
server_key_version: "encryptionkey@0"
I20260925 02:53:34.111929 16493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:53:34.115427 16508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:34.116853 16493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:53:34.117071 16493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/wal
uuid: "d0dffd86c761454b8564bb1a15f24f4c"
format_stamp: "Formatted at 2026-09-25 02:53:34 on dist-test-slave-g1xh"
server_key: "00d978b638e6deae82f1b1f3b96879b3"
server_key_iv: "81e16123dc53dd04094eab7b30277962"
server_key_version: "encryptionkey@0"
I20260925 02:53:34.117285 16493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:34.134383 16493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:34.138765 16493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:34.139074 16493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:34.158106 16493 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.125:45017
I20260925 02:53:34.158145 16560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.125:45017 every 8 connection(s)
I20260925 02:53:34.159533 16493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
I20260925 02:53:34.162948 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 16493
I20260925 02:53:34.163142 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-1/wal/instance
I20260925 02:53:34.163511 16562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:34.163705 2737 external_mini_cluster.cc:1428] Setting key 2af3529c12ccf484a8db9bd993425399
I20260925 02:53:34.170178 16562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304814, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.125@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.172.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.124@KRBTEST.COM" created.
I20260925 02:53:34.191762 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.173010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.2.172.126:41901)
0925 02:53:34.173201 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.173205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.174732 (+ 1527us) server_negotiation.cc:407] Connection header received
0925 02:53:34.176844 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.176851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.176965 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.177086 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.178695 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.179409 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.181047 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.181491 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.181639 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.185334 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.185340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.185343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.185387 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.188285 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.189135 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.189140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.189143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.189268 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.189883 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.189887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.189890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.190204 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.190372 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.191151 (+ 779us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.191390 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:53:34.192840 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.173629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:41901 (local address 127.0.0.1:36584)
0925 02:53:34.174527 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.174576 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:34.176522 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:34.177344 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:34.177353 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:34.177903 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:34.178505 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.178524 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.179609 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.179613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.180866 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.180890 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.181738 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.181745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.181988 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.182713 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:34.182737 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:34.185073 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:34.188469 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.188478 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.188498 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.188983 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.189480 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.189492 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.189496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.189731 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.190360 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:34.190369 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:34.190761 (+ 392us) client_negotiation.cc:769] Sending connection context
0925 02:53:34.191040 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 02:53:34.191607 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":407,"threads_started":1}
I20260925 02:53:34.194716 16562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304814, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.125@KRBTEST.COM for kudu/127.2.172.125@KRBTEST.COM
I20260925 02:53:34.211501 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.195637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51318)
0925 02:53:34.195812 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.195832 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:34.195932 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:34.196911 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:34.196916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:34.197376 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:34.197517 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.197525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.198910 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.198918 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.199914 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.199926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.200774 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.200780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.201005 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.202370 (+ 1365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:34.202397 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:34.205381 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:34.208833 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.208838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.208842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.209220 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.209614 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.209621 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.209624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.209778 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.210417 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:34.210422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:34.210569 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 02:53:34.210716 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 02:53:34.211029 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":1}
I20260925 02:53:34.211604 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.195827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.2.172.125:45017)
0925 02:53:34.196506 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.196514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.196543 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:53:34.196586 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.196592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.196764 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.196918 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.197670 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.198773 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.200055 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.200604 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.200755 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.205626 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.205641 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.205657 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.205692 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.208673 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.209365 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.209374 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.209378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.209504 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.209890 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.209895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.209898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.210274 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.210509 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.210878 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.211177 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":535,"thread_start_us":408,"threads_started":1}
I20260925 02:53:34.214562 16562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:53:34.215893 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.215368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50072)
0925 02:53:34.215626 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.215733 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 02:53:34.216279 16512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.124.keytab.
Entry for principal kudu/127.2.172.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.124.keytab.
I20260925 02:53:34.217847 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.217503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50080)
0925 02:53:34.217665 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.217741 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 02:53:34.218089 16562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:38529: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
I20260925 02:53:34.219106 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.124:38529
--webserver_interface=127.2.172.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.124.keytab
--principal=kudu/127.2.172.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:41901,127.2.172.125:45017,127.2.172.124:38529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:53:34.267483 16562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } attempt: 1
I20260925 02:53:34.268761 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.268286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50088)
0925 02:53:34.268489 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.268604 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 02:53:34.270030 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.269612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50102)
0925 02:53:34.269821 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.269893 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 02:53:34.270320 16562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:38529: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
I20260925 02:53:34.338582 16562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } attempt: 2
I20260925 02:53:34.339908 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.339352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50108)
0925 02:53:34.339664 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.339752 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 02:53:34.341099 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.340729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50124)
0925 02:53:34.340891 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.340961 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 02:53:34.341325 16562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:38529: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
I20260925 02:53:34.355182 16486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } attempt: 5
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304813, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for kudu/127.2.172.125@KRBTEST.COM
I20260925 02:53:34.368527 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.356034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.2.172.125:45017)
0925 02:53:34.356212 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.356218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.356302 (+ 84us) server_negotiation.cc:407] Connection header received
0925 02:53:34.356559 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.356563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.356631 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.356745 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.357534 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.358300 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.359443 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.359951 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.360088 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.363217 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.363222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.363225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.363258 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.365864 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.366525 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.366537 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.366540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.366657 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.367039 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.367045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.367048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.367329 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.367484 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.367981 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.368205 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:53:34.368537 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.355958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51330)
0925 02:53:34.356186 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.356208 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:34.356384 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:34.356773 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:34.356778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:34.357188 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:34.357373 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.357383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.358436 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.358441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.359286 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.359297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.360120 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.360126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.360303 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.360929 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:34.360946 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:34.362914 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:34.366009 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.366016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.366020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.366395 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.366769 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.366774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.366778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.366922 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.367459 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:34.367464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:34.367677 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 02:53:34.367850 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 02:53:34.368140 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 02:53:34.369925 16486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:53:34.371285 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.370794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50138)
0925 02:53:34.370993 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.371100 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 02:53:34.372375 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.372005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50146)
0925 02:53:34.372155 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.372252 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:53:34.372611 16486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:38529: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
I20260925 02:53:34.423873 16486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } attempt: 1
I20260925 02:53:34.425361 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.424673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50148)
0925 02:53:34.424946 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.425115 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 02:53:34.426570 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.426138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50150)
0925 02:53:34.426323 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.426430 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 02:53:34.426810 16486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:38529: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
I20260925 02:53:34.432592 16562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } attempt: 3
I20260925 02:53:34.433657 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.433212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50154)
0925 02:53:34.433395 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.433497 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 02:53:34.434801 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.434300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50156)
0925 02:53:34.434563 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.434650 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260925 02:53:34.435099 16562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:38529: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
I20260925 02:53:34.495079 16486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } attempt: 2
I20260925 02:53:34.496587 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.495867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50158)
0925 02:53:34.496223 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.496347 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 02:53:34.497831 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.497418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50174)
0925 02:53:34.497592 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.497685 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 02:53:34.498207 16486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:38529: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
W20260925 02:53:34.569005 16569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:34.569288 16569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:34.569463 16569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:34.576754 16569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:34.576864 16569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:34.576964 16569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:34.577066 16569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:34.577147 16569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:34.578369 16562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } attempt: 4
I20260925 02:53:34.579663 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.579037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50190)
0925 02:53:34.579300 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.579437 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 02:53:34.580775 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.580387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50196)
0925 02:53:34.580546 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.580644 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:53:34.581035 16562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:38529: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
I20260925 02:53:34.588124 16569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:41901,127.2.172.125:45017,127.2.172.124:38529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.124.keytab
--principal=kudu/127.2.172.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.124:38529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.172.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:34.589527 16569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:34.590912 16569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:34.603262 16574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:34.603355 16575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:34.603964 16577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:34.604387 16569 server_base.cc:1034] running on GCE node
I20260925 02:53:34.604449 16486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } attempt: 3
I20260925 02:53:34.605345 16569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:34.605665 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.605045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50204)
0925 02:53:34.605284 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.605406 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 02:53:34.606767 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.606370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50208)
0925 02:53:34.606587 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.606665 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260925 02:53:34.607023 16486 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:38529: Network error: Client connection negotiation failed: client connection to 127.2.172.124:38529: connect: Connection refused (error 111)
I20260925 02:53:34.607196 16569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:34.608408 16569 hybrid_clock.cc:648] HybridClock initialized: now 1790304814608356 us; error 44 us; skew 500 ppm
I20260925 02:53:34.608772 16569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304814, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:34.612062 16569 init.cc:377] Logged in from keytab as kudu/127.2.172.124@KRBTEST.COM (short username kudu)
I20260925 02:53:34.613799 16569 webserver.cc:460] Webserver started at http://127.2.172.124:36543/ using document root <none> and password file <none>
I20260925 02:53:34.614558 16569 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:34.614670 16569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:34.615005 16569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:34.618335 16569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/data/instance:
uuid: "1d192bc5f6c84741b521d748948d8797"
format_stamp: "Formatted at 2026-09-25 02:53:34 on dist-test-slave-g1xh"
server_key: "29fd8a72cd33874b856db80857dfde18"
server_key_iv: "1d150c543bb069e9546607d92e41eb84"
server_key_version: "encryptionkey@0"
I20260925 02:53:34.619216 16569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/wal/instance:
uuid: "1d192bc5f6c84741b521d748948d8797"
format_stamp: "Formatted at 2026-09-25 02:53:34 on dist-test-slave-g1xh"
server_key: "29fd8a72cd33874b856db80857dfde18"
server_key_iv: "1d150c543bb069e9546607d92e41eb84"
server_key_version: "encryptionkey@0"
I20260925 02:53:34.625029 16569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 02:53:34.628759 16585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:34.630228 16569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:53:34.630407 16569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/wal
uuid: "1d192bc5f6c84741b521d748948d8797"
format_stamp: "Formatted at 2026-09-25 02:53:34 on dist-test-slave-g1xh"
server_key: "29fd8a72cd33874b856db80857dfde18"
server_key_iv: "1d150c543bb069e9546607d92e41eb84"
server_key_version: "encryptionkey@0"
I20260925 02:53:34.630635 16569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:34.653442 16569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:34.657840 16569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:34.658169 16569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:34.676863 16569 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.124:38529
I20260925 02:53:34.676873 16636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.124:38529 every 8 connection(s)
I20260925 02:53:34.678212 16569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
I20260925 02:53:34.681833 16637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:34.684656 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 16569
I20260925 02:53:34.684854 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304596269475-2737-0/minicluster-data/master-2/wal/instance
I20260925 02:53:34.685528 2737 external_mini_cluster.cc:1428] Setting key 03d7a058e719ad61af4792227df5f432
I20260925 02:53:34.689898 16637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.124@KRBTEST.COM
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304814, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.124@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:34.708009 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.692568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.2.172.126:41901)
0925 02:53:34.693063 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.693072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.693521 (+ 449us) server_negotiation.cc:407] Connection header received
0925 02:53:34.693743 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.693753 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.693818 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.693936 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.695336 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.696187 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.697868 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.698292 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.698389 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.702083 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.702089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.702092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.702126 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.704977 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.705689 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.705701 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.705703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.705831 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.706261 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.706267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.706270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.706532 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.706702 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.707334 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.707590 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":259,"threads_started":1}
I20260925 02:53:34.709059 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.692608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:41901 (local address 127.0.0.1:36594)
0925 02:53:34.693375 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.693418 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:34.693610 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:34.694009 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:34.694017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:34.694798 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:34.695177 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.695190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.696342 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.696346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.697717 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.697736 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.698539 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.698546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.698762 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.699444 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:34.699479 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:34.701886 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:34.705103 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.705116 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.705131 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.705552 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.705948 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.705953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.705957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.706141 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.706662 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:34.706671 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:34.706978 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 02:53:34.707242 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 02:53:34.707800 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":426,"threads_started":1}
I20260925 02:53:34.709092 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.687669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.2.172.124:38529)
0925 02:53:34.688465 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.688480 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.688522 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:53:34.690125 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.690148 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.690543 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.690923 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.692558 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.694289 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.696028 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.696820 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.696954 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.701837 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.701850 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.701868 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.701900 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.705077 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.705830 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.705839 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.705843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.706012 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.706412 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.706417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.706421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.707034 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.707342 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.707717 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.707975 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"spinlock_wait_cycles":15104,"thread_start_us":369,"threads_started":1}
I20260925 02:53:34.711423 16637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304814, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.124@KRBTEST.COM for kudu/127.2.172.125@KRBTEST.COM
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:34.726392 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.712275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51338 (local address 127.2.172.125:45017)
0925 02:53:34.712434 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.712438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.712798 (+ 360us) server_negotiation.cc:407] Connection header received
0925 02:53:34.713000 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.713005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.713089 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.713203 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.713891 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.714779 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.716117 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.716754 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.716871 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.720439 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.720444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.720447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.720479 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.723791 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.724555 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.724560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.724563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.724738 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.725111 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.725117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.725121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.725465 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.725622 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.725961 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.726129 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:53:34.726539 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.712300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:45017 (local address 127.0.0.1:51338)
0925 02:53:34.712696 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.712732 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:34.712840 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:34.713231 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:34.713235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:34.713578 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:34.713723 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.713730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.714910 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.714915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.715977 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.715989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.716899 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.716905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.717086 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.718163 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:34.718180 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:34.720235 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:34.723921 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.723924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.723927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.724434 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.724858 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.724861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.724864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.725003 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.725582 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:34.725586 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:34.725712 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 02:53:34.725868 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:53:34.726163 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260925 02:53:34.727504 16637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:53:34.728821 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.712245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.2.172.126:41901)
0925 02:53:34.712545 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.712548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.712629 (+ 81us) server_negotiation.cc:407] Connection header received
0925 02:53:34.712770 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.712778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.712865 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.712970 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.714408 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.715226 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.716871 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.717487 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.717637 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.722300 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.722308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.722312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.722352 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.725730 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.726562 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.726568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.726572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.726872 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.727257 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.727262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.727266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.727540 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.727886 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.728246 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.728494 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304814, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.124@KRBTEST.COM for kudu/127.2.172.124@KRBTEST.COM
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.125@KRBTEST.COM
I20260925 02:53:34.741803 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.728299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.2.172.124:38529)
0925 02:53:34.728468 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.728473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.728649 (+ 176us) server_negotiation.cc:407] Connection header received
0925 02:53:34.728805 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.728812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.728888 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.728986 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.729876 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.730784 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.732546 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.733150 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.733275 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.736403 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.736411 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.736418 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.736460 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.738995 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.739851 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.739856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.739861 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.740009 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.740375 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.740380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.740383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.740646 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.740798 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.741132 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.741480 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":55}
I20260925 02:53:34.741801 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.728279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50236)
0925 02:53:34.728556 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.728575 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:34.728693 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:34.729125 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:34.729128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:34.729546 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:34.729725 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.729736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.730921 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.730927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.732405 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.732419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.733297 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.733302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.733437 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.734039 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:34.734056 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:34.736220 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:34.739134 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.739140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.739144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.739709 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.740145 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.740148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.740150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.740261 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.740762 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:34.740766 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:34.740899 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 02:53:34.741039 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 02:53:34.741332 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260925 02:53:34.745837 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.730628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.2.172.125:45017)
0925 02:53:34.730790 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.730795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.730814 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:53:34.731000 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.731004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.731061 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.731131 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.732716 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.733674 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.735031 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.735446 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.735776 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.740129 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.740134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.740137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.740167 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.743109 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.743936 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.743941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.743945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.744090 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.744469 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.744474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.744478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.744742 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.745014 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.745297 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.745541 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:53:34.747805 16637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797: Bootstrap starting.
I20260925 02:53:34.750728 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:53:34.751513 16637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:34.752615 16637 log.cc:824] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:34.755973 16637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797: No bootstrap required, opened a new log
I20260925 02:53:34.769129 16637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } }
I20260925 02:53:34.769708 16637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:34.769902 16637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d192bc5f6c84741b521d748948d8797, State: Initialized, Role: FOLLOWER
I20260925 02:53:34.770793 16637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } }
I20260925 02:53:34.774386 16637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:34.777926 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.755809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.2.172.125:45017)
0925 02:53:34.756026 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.756035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.756743 (+ 708us) server_negotiation.cc:407] Connection header received
0925 02:53:34.757066 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.757075 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.757182 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.757271 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.760623 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.762096 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.765317 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.766040 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.766153 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.770294 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.770301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.770306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.770353 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.774247 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.775252 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.775260 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.775264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.775424 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.775921 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.775927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.775930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.776372 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.776642 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.777230 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.777472 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:53:34.779659 16651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } } }
I20260925 02:53:34.780157 16651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:34.782883 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.755930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.2.172.124:38529)
0925 02:53:34.756094 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.756100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.763832 (+ 7732us) server_negotiation.cc:407] Connection header received
0925 02:53:34.764327 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.764334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.764438 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.764513 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.766872 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.768031 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.770143 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.771004 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.771138 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.775733 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.775742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.775750 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.775803 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.780045 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.780868 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.780882 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.780886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.781038 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.781438 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.781444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.781447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.781755 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.782019 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.782317 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.782512 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:53:34.784047 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.755385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.2.172.126:41901)
0925 02:53:34.755628 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.755635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.764541 (+ 8906us) server_negotiation.cc:407] Connection header received
0925 02:53:34.764780 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.764788 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.764874 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.764944 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.767330 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.768486 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.770734 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.771431 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.771569 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.775598 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.775605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.775610 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.775654 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.779506 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.780546 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.780555 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.780561 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.780754 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.781313 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.781321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.781327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.781783 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.782071 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.783321 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.783596 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:53:34.784226 16486 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } attempt: 4
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304813, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for kudu/127.2.172.124@KRBTEST.COM
W20260925 02:53:34.794188 16662 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:53:34.794306 16662 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:53:34.793390 16569 master_runner.cc:186] Error getting master registration for 127.2.172.126:41901: OK, instance_id { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" instance_seqno: 1790304813641047 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:53:34.803395 16569 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:53:34.804104 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.784845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50250)
0925 02:53:34.785109 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.785130 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:34.785240 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:34.785702 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:34.785709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:34.786262 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:34.786480 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.786494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.787507 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.787522 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.788963 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.788979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.790246 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.790255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.790430 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.791034 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:34.791057 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:34.796335 (+ 5278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:34.801154 (+ 4819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.801160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.801167 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.801684 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.802106 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.802111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.802114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.802271 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.802978 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:34.802983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:34.803193 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 02:53:34.803349 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:53:34.803698 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 02:53:34.805104 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.784966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.2.172.124:38529)
0925 02:53:34.785110 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.785117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.785180 (+ 63us) server_negotiation.cc:407] Connection header received
0925 02:53:34.785374 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.785379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.785513 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.786240 (+ 727us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.786618 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.787364 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.789142 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.789863 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.789987 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.796540 (+ 6553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.796546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.796552 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.796616 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.800999 (+ 4383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.801817 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.801823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.801827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.801980 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.802400 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.802410 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.802413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.802841 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.803003 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.803743 (+ 740us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.803971 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:53:34.809535 16486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba: Bootstrap starting.
I20260925 02:53:34.812711 16486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:34.813777 16486 log.cc:824] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:34.817692 16486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba: No bootstrap required, opened a new log
I20260925 02:53:34.825570 16486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } }
I20260925 02:53:34.826043 16486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:34.826200 16486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b73825c4a2425bb1a60b9002d4dbba, State: Initialized, Role: FOLLOWER
I20260925 02:53:34.826841 16486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } }
I20260925 02:53:34.827837 16665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } } }
I20260925 02:53:34.828246 16665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:34.829062 16486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:53:34.840040 16676 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:53:34.840245 16676 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:53:34.839359 16419 master_runner.cc:186] Error getting master registration for 127.2.172.125:45017: OK, instance_id { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" instance_seqno: 1790304814152602 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:53:34.850684 16419 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:53:34.862283 16651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:34.862649 16651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } }
I20260925 02:53:34.864223 16651 leader_election.cc:288] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44b73825c4a2425bb1a60b9002d4dbba (127.2.172.126:41901), d0dffd86c761454b8564bb1a15f24f4c (127.2.172.125:45017)
I20260925 02:53:34.865016 16461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d192bc5f6c84741b521d748948d8797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44b73825c4a2425bb1a60b9002d4dbba" is_pre_election: true
I20260925 02:53:34.865779 16461 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d192bc5f6c84741b521d748948d8797 in term 0.
I20260925 02:53:34.866391 16589 leader_election.cc:302] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d192bc5f6c84741b521d748948d8797, 44b73825c4a2425bb1a60b9002d4dbba; no voters:
I20260925 02:53:34.866791 16651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:34.866943 16651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:34.867059 16651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:34.870424 16651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } }
I20260925 02:53:34.871590 16461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d192bc5f6c84741b521d748948d8797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44b73825c4a2425bb1a60b9002d4dbba"
I20260925 02:53:34.871814 16461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:34.875411 16461 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d192bc5f6c84741b521d748948d8797 in term 1.
I20260925 02:53:34.875943 16589 leader_election.cc:302] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d192bc5f6c84741b521d748948d8797, 44b73825c4a2425bb1a60b9002d4dbba; no voters:
I20260925 02:53:34.878444 16651 leader_election.cc:288] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [CANDIDATE]: Term 1 election: Requested vote from peers 44b73825c4a2425bb1a60b9002d4dbba (127.2.172.126:41901), d0dffd86c761454b8564bb1a15f24f4c (127.2.172.125:45017)
I20260925 02:53:34.878695 16651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:34.878846 16651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [term 1 LEADER]: Becoming Leader. State: Replica: 1d192bc5f6c84741b521d748948d8797, State: Running, Role: LEADER
W20260925 02:53:34.879806 16440 tablet.cc:2369] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:34.879457 16651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } }
I20260925 02:53:34.880254 16562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } attempt: 5
I20260925 02:53:34.896205 16678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d192bc5f6c84741b521d748948d8797. Latest consensus state: current_term: 1 leader_uuid: "1d192bc5f6c84741b521d748948d8797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } } }
Sep 25 02:53:34 dist-test-slave-g1xh krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304814, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.125@KRBTEST.COM for kudu/127.2.172.124@KRBTEST.COM
I20260925 02:53:34.896663 16678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:34.904654 16680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:34.906694 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.880821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:38529 (local address 127.0.0.1:50260)
0925 02:53:34.881010 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.881030 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:34.881143 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:34.881716 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:34.881723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:34.882307 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:34.882527 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.882541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.884000 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.884005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.885439 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.885461 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.887654 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.887663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.888075 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.893162 (+ 5087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:34.893216 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:34.898591 (+ 5375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:34.903314 (+ 4723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.903322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.903326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.903973 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.904403 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:34.904408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:34.904413 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:34.904602 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:34.905565 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:34.905573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:34.905774 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 02:53:34.905964 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 02:53:34.906293 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 02:53:34.907797 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.880856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.2.172.124:38529)
0925 02:53:34.881403 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.881411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.881433 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:53:34.881496 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.881504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.881581 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.881678 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.882670 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.883845 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.885682 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.886659 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.886806 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.899029 (+ 12223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:34.899043 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:34.899050 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:34.899107 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:34.903137 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.904103 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.904110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.904113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.904270 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:34.904752 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:34.904759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:34.904764 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:34.905253 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:34.905430 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:34.907053 (+ 1623us) server_negotiation.cc:299] Negotiation successful
0925 02:53:34.907334 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 02:53:34.910334 16591 tablet.cc:2369] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:34.911521 16680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:34.913893 16562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c: Bootstrap starting.
I20260925 02:53:34.917884 16562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:34.919238 16562 log.cc:824] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:34.937491 16461 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [term 1 FOLLOWER]: Refusing update from remote peer 1d192bc5f6c84741b521d748948d8797: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:34.937649 16562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c: No bootstrap required, opened a new log
I20260925 02:53:34.938601 16678 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:34.966274 16665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d192bc5f6c84741b521d748948d8797. Latest consensus state: current_term: 1 leader_uuid: "1d192bc5f6c84741b521d748948d8797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } } }
I20260925 02:53:34.966624 16665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:34.968528 16681 mvcc.cc:205] Tried to move back new op lower bound from 7333088521970442240 to 7333088521764818944. Current Snapshot: MvccSnapshot[applied={T|T < 7333088521970442240}]
I20260925 02:53:34.968731 16562 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } }
I20260925 02:53:34.969696 16562 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:34.969972 16562 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dffd86c761454b8564bb1a15f24f4c, State: Initialized, Role: FOLLOWER
I20260925 02:53:34.971468 16562 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } }
I20260925 02:53:34.983310 16683 mvcc.cc:205] Tried to move back new op lower bound from 7333088521970442240 to 7333088521764818944. Current Snapshot: MvccSnapshot[applied={T|T < 7333088521970442240}]
I20260925 02:53:34.984236 16687 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } } }
I20260925 02:53:34.984756 16687 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:34.989924 16535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d192bc5f6c84741b521d748948d8797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dffd86c761454b8564bb1a15f24f4c"
I20260925 02:53:34.990476 16536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d192bc5f6c84741b521d748948d8797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dffd86c761454b8564bb1a15f24f4c" is_pre_election: true
I20260925 02:53:34.991607 16534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:34.994088 16562 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:34.994812 16665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d192bc5f6c84741b521d748948d8797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } } }
I20260925 02:53:34.995085 16665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:34.995479 16534 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [term 1 FOLLOWER]: Refusing update from remote peer 1d192bc5f6c84741b521d748948d8797: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:34.999739 16678 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:35.003623 16680 catalog_manager.cc:1347] Generated new cluster ID: b08ac8eba6eb4c8c90853f1833282420
I20260925 02:53:35.003729 16680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:35.004837 16678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d192bc5f6c84741b521d748948d8797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } } }
I20260925 02:53:35.008659 16678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:35.033757 16651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d192bc5f6c84741b521d748948d8797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } } }
I20260925 02:53:35.034245 16651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:35.034137 16687 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d192bc5f6c84741b521d748948d8797. Latest consensus state: current_term: 1 leader_uuid: "1d192bc5f6c84741b521d748948d8797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } } }
I20260925 02:53:35.034498 16687 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:35.037467 16680 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:35.039299 16680 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:35.037144 16693 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d192bc5f6c84741b521d748948d8797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b73825c4a2425bb1a60b9002d4dbba" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 41901 } } peers { permanent_uuid: "d0dffd86c761454b8564bb1a15f24f4c" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 45017 } } peers { permanent_uuid: "1d192bc5f6c84741b521d748948d8797" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 38529 } } }
I20260925 02:53:35.040352 16693 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:35.064491 16680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d192bc5f6c84741b521d748948d8797: Generated new TSK 0
I20260925 02:53:35.065588 16680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:53:35.123667 16704 catalog_manager.cc:1259] Loaded cluster ID: b08ac8eba6eb4c8c90853f1833282420
I20260925 02:53:35.124820 16704 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c: loading cluster ID for follower catalog manager: success
I20260925 02:53:35.131295 16704 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c: acquiring CA information for follower catalog manager: success
I20260925 02:53:35.142549 16704 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0dffd86c761454b8564bb1a15f24f4c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:53:35.842931 16676 catalog_manager.cc:1259] Loaded cluster ID: b08ac8eba6eb4c8c90853f1833282420
I20260925 02:53:35.843104 16676 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba: loading cluster ID for follower catalog manager: success
I20260925 02:53:35.846060 16676 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba: acquiring CA information for follower catalog manager: success
I20260925 02:53:35.848380 16676 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44b73825c4a2425bb1a60b9002d4dbba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:53:36.217926 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 16419
I20260925 02:53:36.220364 16481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:36.350011 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 16419
I20260925 02:53:36.376394 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 16493
I20260925 02:53:36.377883 16556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:36.504717 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 16493
I20260925 02:53:36.531731 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 16569
I20260925 02:53:36.533121 16632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:36.660559 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 16569
2026-09-25T02:53:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4867 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:53:36 dist-test-slave-g1xh krb5kdc[16715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:53:36 dist-test-slave-g1xh krb5kdc[16715](info): set up 2 sockets
Sep 25 02:53:36 dist-test-slave-g1xh krb5kdc[16715](info): commencing operation
krb5kdc: starting...
W20260925 02:53:37.851612 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:53:38 dist-test-slave-g1xh krb5kdc[16715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:53:38.119248 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:36857
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:45397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:36857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:38.461424 16728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:38.461740 16728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:38.461906 16728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:38.469933 16728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:38.470045 16728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:38.470144 16728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:38.470244 16728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:38.470337 16728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:38.481194 16728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:45397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:36857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:36857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:38.482723 16728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:38.484272 16728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:38.496592 16733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:38.496623 16736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:38.496647 16734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:38.497792 16728 server_base.cc:1034] running on GCE node
I20260925 02:53:38.498708 16728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:38.500370 16728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:38.501626 16728 hybrid_clock.cc:648] HybridClock initialized: now 1790304818501590 us; error 56 us; skew 500 ppm
I20260925 02:53:38.502004 16728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:38 dist-test-slave-g1xh krb5kdc[16715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304818, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:38.505602 16728 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:53:38.507372 16728 webserver.cc:460] Webserver started at http://127.2.172.126:42265/ using document root <none> and password file <none>
I20260925 02:53:38.508240 16728 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:38.508405 16728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:38.508761 16728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:38.512076 16728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "9b8ae3dfda01424ab78fb2900b914c47"
format_stamp: "Formatted at 2026-09-25 02:53:38 on dist-test-slave-g1xh"
server_key: "1e2434d52f48f6993594616e616b9559"
server_key_iv: "0219cd1c13cc2b3898e91f35a9646708"
server_key_version: "encryptionkey@0"
I20260925 02:53:38.512892 16728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "9b8ae3dfda01424ab78fb2900b914c47"
format_stamp: "Formatted at 2026-09-25 02:53:38 on dist-test-slave-g1xh"
server_key: "1e2434d52f48f6993594616e616b9559"
server_key_iv: "0219cd1c13cc2b3898e91f35a9646708"
server_key_version: "encryptionkey@0"
I20260925 02:53:38.518904 16728 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260925 02:53:38.522760 16743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:38.524416 16728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:53:38.524638 16728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "9b8ae3dfda01424ab78fb2900b914c47"
format_stamp: "Formatted at 2026-09-25 02:53:38 on dist-test-slave-g1xh"
server_key: "1e2434d52f48f6993594616e616b9559"
server_key_iv: "0219cd1c13cc2b3898e91f35a9646708"
server_key_version: "encryptionkey@0"
I20260925 02:53:38.524899 16728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:38.547200 16728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:38.551921 16728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:38.552192 16728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:38.572090 16728 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:36857
I20260925 02:53:38.572113 16794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:36857 every 8 connection(s)
I20260925 02:53:38.573633 16728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:38.576166 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 16728
I20260925 02:53:38.576381 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:53:38.577127 2737 external_mini_cluster.cc:1428] Setting key 340e1eff0562dcb31fbe4b444b41bf73
I20260925 02:53:38.577732 16795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:38.592159 16795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47: Bootstrap starting.
I20260925 02:53:38.595263 16795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:38.596338 16795 log.cc:824] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:38.599287 16795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47: No bootstrap required, opened a new log
Sep 25 02:53:38 dist-test-slave-g1xh krb5kdc[16715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:38.610322 16795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8ae3dfda01424ab78fb2900b914c47" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36857 } }
I20260925 02:53:38.610809 16795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:38.610947 16795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8ae3dfda01424ab78fb2900b914c47, State: Initialized, Role: FOLLOWER
I20260925 02:53:38.611819 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:38.579071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.2.172.126:36857)
0925 02:53:38.579952 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:38.579967 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:38.580017 (+ 50us) server_negotiation.cc:407] Connection header received
0925 02:53:38.582531 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:38.582560 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:38.583066 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:38.583517 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:38.584940 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.586903 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.588457 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.589602 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.589798 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:38.602092 (+ 12294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:38.602107 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:38.602135 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:38.602183 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:38.606512 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:38.607469 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:38.607482 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:38.607493 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:38.607781 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:38.608280 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:38.608286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:38.608289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:38.609093 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:38.609396 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:38.609804 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 02:53:38.610267 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":422,"threads_started":1}
I20260925 02:53:38.611737 16795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8ae3dfda01424ab78fb2900b914c47" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36857 } }
I20260925 02:53:38.612066 16795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:38.612228 16795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:38.612468 16795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:38.615444 16795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8ae3dfda01424ab78fb2900b914c47" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36857 } }
I20260925 02:53:38.616065 16795 leader_election.cc:302] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b8ae3dfda01424ab78fb2900b914c47; no voters:
I20260925 02:53:38.616463 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:53:38.616820 16795 leader_election.cc:288] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:38.616935 16800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:38.619730 16795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:38.624002 16800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [term 1 LEADER]: Becoming Leader. State: Replica: 9b8ae3dfda01424ab78fb2900b914c47, State: Running, Role: LEADER
I20260925 02:53:38.624712 16800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8ae3dfda01424ab78fb2900b914c47" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36857 } }
I20260925 02:53:38.628396 16802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b8ae3dfda01424ab78fb2900b914c47. Latest consensus state: current_term: 1 leader_uuid: "9b8ae3dfda01424ab78fb2900b914c47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8ae3dfda01424ab78fb2900b914c47" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36857 } } }
I20260925 02:53:38.628862 16802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:38.629367 16801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b8ae3dfda01424ab78fb2900b914c47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8ae3dfda01424ab78fb2900b914c47" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36857 } } }
I20260925 02:53:38.629830 16801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:38.637253 16812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:38.641781 16812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:38.650465 16812 catalog_manager.cc:1347] Generated new cluster ID: 0828ba2fb95f4d2eb9783049655e864f
I20260925 02:53:38.650626 16812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:38.664042 16812 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:38.665459 16812 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:38.671877 16812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b8ae3dfda01424ab78fb2900b914c47: Generated new TSK 0
I20260925 02:53:38.673081 16812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:53:38.729921 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 16728
I20260925 02:53:38.731138 16790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:38.833496 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 16728
2026-09-25T02:53:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2159 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:53:38 dist-test-slave-g1xh krb5kdc[16823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:53:38 dist-test-slave-g1xh krb5kdc[16823](info): set up 2 sockets
Sep 25 02:53:38 dist-test-slave-g1xh krb5kdc[16823](info): commencing operation
krb5kdc: starting...
W20260925 02:53:39.962266 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:53:40 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:53:40.228775 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:42799
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:42799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:40.565598 16836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:40.565923 16836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:40.566016 16836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:40.574082 16836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:40.574252 16836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:40.574309 16836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:40.574364 16836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:40.574405 16836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:40.585326 16836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:42799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:42799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:40.586777 16836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:40.588145 16836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:40.601428 16841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:40.601567 16844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:40.601475 16842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:40.602555 16836 server_base.cc:1034] running on GCE node
I20260925 02:53:40.603694 16836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:40.605331 16836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:40.606607 16836 hybrid_clock.cc:648] HybridClock initialized: now 1790304820606541 us; error 46 us; skew 500 ppm
I20260925 02:53:40.607036 16836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:40 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304820, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:40.610615 16836 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:53:40.612429 16836 webserver.cc:460] Webserver started at http://127.2.172.126:36787/ using document root <none> and password file <none>
I20260925 02:53:40.613217 16836 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:40.613381 16836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:40.613727 16836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:40.617187 16836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "538ffedac61748c4825c95e059559e98"
format_stamp: "Formatted at 2026-09-25 02:53:40 on dist-test-slave-g1xh"
server_key: "474e9e4346243a4cb2899f9e07ac8daf"
server_key_iv: "22e1cac630bbdf3a6564aabfeef222bd"
server_key_version: "encryptionkey@0"
I20260925 02:53:40.618075 16836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "538ffedac61748c4825c95e059559e98"
format_stamp: "Formatted at 2026-09-25 02:53:40 on dist-test-slave-g1xh"
server_key: "474e9e4346243a4cb2899f9e07ac8daf"
server_key_iv: "22e1cac630bbdf3a6564aabfeef222bd"
server_key_version: "encryptionkey@0"
I20260925 02:53:40.624102 16836 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 02:53:40.627841 16851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:40.629513 16836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:53:40.629750 16836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "538ffedac61748c4825c95e059559e98"
format_stamp: "Formatted at 2026-09-25 02:53:40 on dist-test-slave-g1xh"
server_key: "474e9e4346243a4cb2899f9e07ac8daf"
server_key_iv: "22e1cac630bbdf3a6564aabfeef222bd"
server_key_version: "encryptionkey@0"
I20260925 02:53:40.629983 16836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:40.650031 16836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:40.654699 16836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:40.655025 16836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:40.674521 16836 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:42799
I20260925 02:53:40.674615 16902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:42799 every 8 connection(s)
I20260925 02:53:40.675976 16836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:40.679977 16903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:40.685206 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 16836
I20260925 02:53:40.685469 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:53:40.686105 2737 external_mini_cluster.cc:1428] Setting key 6d64b4696c0e106698a3b5b42d86a785
I20260925 02:53:40.692378 16903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Bootstrap starting.
I20260925 02:53:40.696189 16903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:40.697314 16903 log.cc:824] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:40.700587 16903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: No bootstrap required, opened a new log
Sep 25 02:53:40 dist-test-slave-g1xh krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:40.711365 16903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } }
I20260925 02:53:40.711879 16903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:40.712019 16903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538ffedac61748c4825c95e059559e98, State: Initialized, Role: FOLLOWER
I20260925 02:53:40.712770 16903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } }
I20260925 02:53:40.713042 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:40.688466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52372 (local address 127.2.172.126:42799)
0925 02:53:40.689468 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:40.689484 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:40.689558 (+ 74us) server_negotiation.cc:407] Connection header received
0925 02:53:40.691999 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:40.692033 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:40.692568 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:40.693020 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:40.694411 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:40.696415 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:40.697903 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:40.698747 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:40.698935 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:40.703414 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:40.703428 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:40.703456 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:40.703513 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:40.707768 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:40.708598 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:40.708612 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:40.708621 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:40.708862 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:40.709348 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:40.709353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:40.709357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:40.710161 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:40.710487 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:40.711005 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 02:53:40.711473 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":489,"threads_started":1}
I20260925 02:53:40.713073 16903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:40.713201 16903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:40.713408 16903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:40.716430 16903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } }
I20260925 02:53:40.716903 16903 leader_election.cc:302] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 538ffedac61748c4825c95e059559e98; no voters:
I20260925 02:53:40.717578 16903 leader_election.cc:288] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:40.717768 16908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:40.718475 16908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 1 LEADER]: Becoming Leader. State: Replica: 538ffedac61748c4825c95e059559e98, State: Running, Role: LEADER
I20260925 02:53:40.719115 16908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } }
I20260925 02:53:40.719914 16903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:40.722565 16910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 538ffedac61748c4825c95e059559e98. Latest consensus state: current_term: 1 leader_uuid: "538ffedac61748c4825c95e059559e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } } }
I20260925 02:53:40.722597 16909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "538ffedac61748c4825c95e059559e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } } }
I20260925 02:53:40.722937 16910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:40.723016 16909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:40.724491 16916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:40.729868 16916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:40.738803 16916 catalog_manager.cc:1347] Generated new cluster ID: fe4fa6fe738f496b9ca241bddf2831b5
I20260925 02:53:40.738929 16916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:40.747229 16916 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:40.748775 16916 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:40.757975 16916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Generated new TSK 0
I20260925 02:53:40.759061 16916 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:53:40.827064 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:42799
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:41.178522 16929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:41.178846 16929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:41.179005 16929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:41.186684 16929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:41.186810 16929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:41.186992 16929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:53:41.197896 16929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:41.199390 16929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:41.200860 16929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:41.216336 16934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:41.216410 16935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:41.217562 16937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:41.217566 16929 server_base.cc:1034] running on GCE node
I20260925 02:53:41.219522 16929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:41.221163 16929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:41.222682 16929 hybrid_clock.cc:648] HybridClock initialized: now 1790304821222597 us; error 87 us; skew 500 ppm
I20260925 02:53:41.223044 16929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:41 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304821, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:41.227398 16929 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:53:41.229522 16929 webserver.cc:460] Webserver started at http://127.2.172.65:38181/ using document root <none> and password file <none>
I20260925 02:53:41.230659 16929 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:41.230873 16929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:41.231230 16929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:41.234817 16929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "94db12ef415843c0ad414ca2fdd6df2d"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-g1xh"
server_key: "1be9fafad0527f00e401bcf0bc844422"
server_key_iv: "c0bd2c670771298c8b92fedad4eb8865"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.235824 16929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "94db12ef415843c0ad414ca2fdd6df2d"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-g1xh"
server_key: "1be9fafad0527f00e401bcf0bc844422"
server_key_iv: "c0bd2c670771298c8b92fedad4eb8865"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.242008 16929 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:53:41.246138 16944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.247730 16929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:53:41.247922 16929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "94db12ef415843c0ad414ca2fdd6df2d"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-g1xh"
server_key: "1be9fafad0527f00e401bcf0bc844422"
server_key_iv: "c0bd2c670771298c8b92fedad4eb8865"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.248155 16929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:41.268038 16929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:41.272585 16929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:41.272904 16929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:41.273947 16929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:41.275419 16929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:41.275529 16929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.275704 16929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:41.275786 16929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.310523 16929 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:38737
I20260925 02:53:41.310575 17056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:38737 every 8 connection(s)
I20260925 02:53:41.311944 16929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:41.317334 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 16929
I20260925 02:53:41.317576 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:41.318425 2737 external_mini_cluster.cc:1428] Setting key 31c3d0d0fa78552ace2b96da96ae6e08
Sep 25 02:53:41 dist-test-slave-g1xh krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304821, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:41.337193 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.314615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:51069 (local address 127.2.172.126:42799)
0925 02:53:41.315101 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:41.315106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:41.316016 (+ 910us) server_negotiation.cc:407] Connection header received
0925 02:53:41.318566 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:41.318576 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:41.318673 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:41.318775 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:41.320753 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.321559 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.324315 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.324848 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.324970 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:41.329664 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:41.329669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:41.329673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:41.329708 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:41.333397 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:41.334268 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:41.334277 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:41.334283 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:41.334436 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:41.334907 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:41.334915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:41.334920 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:41.335371 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:41.335527 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:41.336370 (+ 843us) server_negotiation.cc:299] Negotiation successful
0925 02:53:41.336640 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":258,"threads_started":1}
I20260925 02:53:41.337850 17060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.315071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:51069)
0925 02:53:41.315847 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:41.315892 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:41.318158 (+ 2266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:41.319071 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:41.319085 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:41.319748 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:41.320517 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:41.320544 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.321735 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.321744 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:41.324098 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:41.324119 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.325112 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.325122 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:41.325473 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:41.326272 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:41.326312 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:41.329381 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:41.333580 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:41.333587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:41.333601 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:41.334130 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:41.334549 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:41.334552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:41.334554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:41.334776 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:41.335521 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:41.335528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:41.335934 (+ 406us) client_negotiation.cc:769] Sending connection context
0925 02:53:41.336206 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 02:53:41.336656 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":313,"threads_started":1}
I20260925 02:53:41.339499 17057 heartbeater.cc:347] Connected to a master server at 127.2.172.126:42799
I20260925 02:53:41.339865 17057 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:41.340628 17057 heartbeater.cc:511] Master 127.2.172.126:42799 requested a full tablet report, sending...
I20260925 02:53:41.342870 16868 ts_manager.cc:194] Registered new tserver with Master: 94db12ef415843c0ad414ca2fdd6df2d (127.2.172.65:38737)
I20260925 02:53:41.344393 16868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:51069
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:53:41.378968 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:42799
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:41.732703 17063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:41.733052 17063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:41.733222 17063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:41.740963 17063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:41.741148 17063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:41.741330 17063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:53:41.751514 17063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:41.753155 17063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:41.754565 17063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:41.768841 17068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:41.768790 17069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:41.768812 17071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:41.769697 17063 server_base.cc:1034] running on GCE node
I20260925 02:53:41.770339 17063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:41.771361 17063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:41.772576 17063 hybrid_clock.cc:648] HybridClock initialized: now 1790304821772530 us; error 39 us; skew 500 ppm
I20260925 02:53:41.772933 17063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:41 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304821, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:41.776247 17063 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:53:41.777701 17063 webserver.cc:460] Webserver started at http://127.2.172.66:42619/ using document root <none> and password file <none>
I20260925 02:53:41.778414 17063 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:41.778600 17063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:41.778913 17063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:41.781992 17063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-g1xh"
server_key: "d216ab606a46290afd34d36820f009ed"
server_key_iv: "ee74c953a60988edca05ca84f177487c"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.782735 17063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-g1xh"
server_key: "d216ab606a46290afd34d36820f009ed"
server_key_iv: "ee74c953a60988edca05ca84f177487c"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.788235 17063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 02:53:41.793635 17078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.796312 17063 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:53:41.796932 17063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-g1xh"
server_key: "d216ab606a46290afd34d36820f009ed"
server_key_iv: "ee74c953a60988edca05ca84f177487c"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.797358 17063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:41.819005 17063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:41.824080 17063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:41.824410 17063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:41.825583 17063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:41.827114 17063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:41.827231 17063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.827397 17063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:41.827484 17063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.861939 17063 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:39675
I20260925 02:53:41.862032 17191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:39675 every 8 connection(s)
I20260925 02:53:41.863268 17063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:41.865157 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 17063
I20260925 02:53:41.865365 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:53:41.865816 2737 external_mini_cluster.cc:1428] Setting key f83c814a406c0320d71ef9420ada23c7
Sep 25 02:53:41 dist-test-slave-g1xh krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304821, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:41.884346 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.866147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:47995 (local address 127.2.172.126:42799)
0925 02:53:41.866631 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:41.866638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:41.867809 (+ 1171us) server_negotiation.cc:407] Connection header received
0925 02:53:41.869726 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:41.869732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:41.869825 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:41.869949 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:41.871488 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.872261 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.873863 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.874281 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.874411 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:41.878481 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:41.878486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:41.878489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:41.878523 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:41.881137 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:41.881942 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:41.881949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:41.881952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:41.882093 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:41.882522 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:41.882529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:41.882535 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:41.882817 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:41.882975 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:41.883723 (+ 748us) server_negotiation.cc:299] Negotiation successful
0925 02:53:41.883956 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260925 02:53:41.885289 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.866787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:47995)
0925 02:53:41.867636 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:41.867685 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:41.869431 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:41.870219 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:41.870228 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:41.870740 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:41.871305 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:41.871322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.872424 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.872435 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:41.873710 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:41.873723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.874528 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.874535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:41.874825 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:41.875601 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:41.875641 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:41.878245 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:41.881338 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:41.881347 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:41.881366 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:41.881815 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:41.882209 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:41.882214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:41.882218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:41.882408 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:41.882958 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:41.882966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:41.883313 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 02:53:41.883624 (+ 311us) client_negotiation.cc:240] Negotiation successful
0925 02:53:41.884160 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":344,"threads_started":1}
I20260925 02:53:41.886808 17192 heartbeater.cc:347] Connected to a master server at 127.2.172.126:42799
I20260925 02:53:41.887171 17192 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:41.888154 17192 heartbeater.cc:511] Master 127.2.172.126:42799 requested a full tablet report, sending...
I20260925 02:53:41.889953 16868 ts_manager.cc:194] Registered new tserver with Master: fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675)
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
I20260925 02:53:41.891078 16868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:47995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:53:41.922434 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:42799
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:42.260807 17198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:42.261094 17198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:42.261260 17198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:42.268682 17198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:42.268800 17198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:42.268989 17198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:53:42.278774 17198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:42.280148 17198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:42.281464 17198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:42.295022 17206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:42.295065 17204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:42.295065 17203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:42.296200 17198 server_base.cc:1034] running on GCE node
I20260925 02:53:42.296855 17198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:42.297901 17198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:42.299113 17198 hybrid_clock.cc:648] HybridClock initialized: now 1790304822299071 us; error 36 us; skew 500 ppm
I20260925 02:53:42.299455 17198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:42 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304822, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:42.303078 17198 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:53:42.304633 17198 webserver.cc:460] Webserver started at http://127.2.172.67:35813/ using document root <none> and password file <none>
I20260925 02:53:42.305371 17198 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:42.305529 17198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:42.305843 17198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:42.308917 17198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e"
format_stamp: "Formatted at 2026-09-25 02:53:42 on dist-test-slave-g1xh"
server_key: "693c1955b114b37c5252973dcd95c1c4"
server_key_iv: "d7393604a2a6e9224860fcde41a03e1d"
server_key_version: "encryptionkey@0"
I20260925 02:53:42.309747 17198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e"
format_stamp: "Formatted at 2026-09-25 02:53:42 on dist-test-slave-g1xh"
server_key: "693c1955b114b37c5252973dcd95c1c4"
server_key_iv: "d7393604a2a6e9224860fcde41a03e1d"
server_key_version: "encryptionkey@0"
I20260925 02:53:42.316810 17198 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260925 02:53:42.320765 17213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:42.322369 17198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:53:42.322580 17198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e"
format_stamp: "Formatted at 2026-09-25 02:53:42 on dist-test-slave-g1xh"
server_key: "693c1955b114b37c5252973dcd95c1c4"
server_key_iv: "d7393604a2a6e9224860fcde41a03e1d"
server_key_version: "encryptionkey@0"
I20260925 02:53:42.322819 17198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:42.347891 17057 heartbeater.cc:503] Master 127.2.172.126:42799 was elected leader, sending a full tablet report...
I20260925 02:53:42.364396 17198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:42.369226 17198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:42.369582 17198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:42.370690 17198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:42.372185 17198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:42.372288 17198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:42.372450 17198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:42.372525 17198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:42.407822 17198 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:43437
I20260925 02:53:42.407866 17325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:43437 every 8 connection(s)
I20260925 02:53:42.409158 17198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:42.417867 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 17198
I20260925 02:53:42.418072 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:42.418560 2737 external_mini_cluster.cc:1428] Setting key 4316337f9b3e99567878bd17e7bfebee
Sep 25 02:53:42 dist-test-slave-g1xh krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304822, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:42.430420 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.412098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:41373 (local address 127.2.172.126:42799)
0925 02:53:42.412602 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.412606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.413348 (+ 742us) server_negotiation.cc:407] Connection header received
0925 02:53:42.415286 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.415294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.415404 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.415534 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:42.417108 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.417978 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.419872 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.420460 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.420606 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.424244 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:42.424249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:42.424252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:42.424286 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:42.427147 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:42.428001 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:42.428005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:42.428007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:42.428169 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:42.428611 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:42.428616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:42.428619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:42.428892 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:42.429070 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:42.429798 (+ 728us) server_negotiation.cc:299] Negotiation successful
0925 02:53:42.430035 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":251,"threads_started":1}
I20260925 02:53:42.431432 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.412395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:41373)
0925 02:53:42.413169 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:42.413216 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:42.414945 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:42.415800 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:42.415809 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:42.416332 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:42.416928 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:42.416946 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.418135 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.418139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:42.419730 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:42.419743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.420694 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.420701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:42.420943 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.421667 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:42.421690 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:42.424003 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:42.427293 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:42.427305 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:42.427323 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:42.427851 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:42.428277 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:42.428283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:42.428286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:42.428486 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:42.429051 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:42.429059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:42.429401 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 02:53:42.429684 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 02:53:42.430225 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":351,"threads_started":1}
I20260925 02:53:42.433156 17326 heartbeater.cc:347] Connected to a master server at 127.2.172.126:42799
I20260925 02:53:42.433506 17326 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:42.434430 17326 heartbeater.cc:511] Master 127.2.172.126:42799 requested a full tablet report, sending...
I20260925 02:53:42.435938 16868 ts_manager.cc:194] Registered new tserver with Master: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437)
I20260925 02:53:42.436782 16868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:41373
I20260925 02:53:42.447067 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:53:42.465735 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.452099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.2.172.126:42799)
0925 02:53:42.452414 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.452419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.452846 (+ 427us) server_negotiation.cc:407] Connection header received
0925 02:53:42.452982 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.452985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.453058 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.453151 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:42.454655 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.455446 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.456939 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.457339 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.457450 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.460195 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:42.460200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:42.460204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:42.460241 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:42.463007 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:42.463770 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:42.463774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:42.463776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:42.463895 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:42.464308 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:42.464312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:42.464314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:42.464714 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:42.464957 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:42.465229 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 02:53:42.465475 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260925 02:53:42.470422 16868 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:53:42.477999 16868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:42.515724 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.491712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:43437 (local address 127.0.0.1:35332)
0925 02:53:42.504516 (+ 12804us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:42.504563 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:42.504818 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:42.506499 (+ 1681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:42.506510 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:42.506568 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:42.507429 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:42.507466 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.509604 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.509611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:42.512459 (+ 2848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:42.512474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.514492 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.514499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:42.514732 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.514790 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 02:53:42.514912 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 02:53:42.515170 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12493,"mutex_wait_us":71,"thread_start_us":14943,"threads_started":1}
I20260925 02:53:42.516247 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.492043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39675 (local address 127.0.0.1:52156)
0925 02:53:42.506658 (+ 14615us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:42.506685 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:42.506936 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:42.507869 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:42.507873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:42.507897 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:42.508372 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:42.508388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.510270 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.510275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:42.513173 (+ 2898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:42.513189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.515280 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.515286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:42.515481 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.515522 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 02:53:42.515648 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 02:53:42.515816 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14397,"mutex_wait_us":39,"thread_start_us":14974,"threads_started":1}
I20260925 02:53:42.516256 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.492100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.2.172.67:43437)
0925 02:53:42.505654 (+ 13554us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.505665 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.505710 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:53:42.505827 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.505836 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.506158 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.506812 (+ 654us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:42.507946 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.509451 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.512642 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.513950 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.514097 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.514346 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:42.515685 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0925 02:53:42.515875 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13341,"thread_start_us":13162,"threads_started":1}
I20260925 02:53:42.516923 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.492801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.2.172.66:39675)
0925 02:53:42.506481 (+ 13680us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.506491 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.506877 (+ 386us) server_negotiation.cc:407] Connection header received
0925 02:53:42.507182 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.507190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.507445 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.507677 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:42.508587 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.510099 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.513364 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.514793 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.514936 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.515108 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:42.516513 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0925 02:53:42.516717 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13493,"thread_start_us":13379,"threads_started":1}
I20260925 02:53:42.516353 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.492645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:38737 (local address 127.0.0.1:55116)
0925 02:53:42.507306 (+ 14661us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:42.507332 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:42.507470 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:42.508193 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:42.508198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:42.508231 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:42.508683 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:42.508696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.510618 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.510624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:42.513448 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:42.513458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.515366 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.515377 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:42.515598 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.515637 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:53:42.515734 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 02:53:42.515868 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14532,"mutex_wait_us":92,"thread_start_us":11368,"threads_started":1}
I20260925 02:53:42.517936 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.493368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.2.172.65:38737)
0925 02:53:42.506039 (+ 12671us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.506047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.507627 (+ 1580us) server_negotiation.cc:407] Connection header received
0925 02:53:42.507736 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.507743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.507962 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.508157 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:42.508882 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.510446 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.513651 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.515116 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.515287 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.515522 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:42.516719 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0925 02:53:42.516896 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12492,"thread_start_us":12323,"threads_started":1}
I20260925 02:53:42.520750 17127 tablet_service.cc:1467] Processing CreateTablet for tablet 60e181c0876d41c0a7f9d2b8db1cab32 (DEFAULT_TABLE table=test-table [id=0804864e29b748b0a91ac8ab0ef28965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:42.520750 17261 tablet_service.cc:1467] Processing CreateTablet for tablet 60e181c0876d41c0a7f9d2b8db1cab32 (DEFAULT_TABLE table=test-table [id=0804864e29b748b0a91ac8ab0ef28965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:42.522367 17127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e181c0876d41c0a7f9d2b8db1cab32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:42.522364 17261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e181c0876d41c0a7f9d2b8db1cab32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:42.522128 16992 tablet_service.cc:1467] Processing CreateTablet for tablet 60e181c0876d41c0a7f9d2b8db1cab32 (DEFAULT_TABLE table=test-table [id=0804864e29b748b0a91ac8ab0ef28965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:42.523530 16992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e181c0876d41c0a7f9d2b8db1cab32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:42.536556 17345 tablet_bootstrap.cc:492] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d: Bootstrap starting.
I20260925 02:53:42.540340 17346 tablet_bootstrap.cc:492] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604: Bootstrap starting.
I20260925 02:53:42.541683 17345 tablet_bootstrap.cc:654] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:42.543102 17345 log.cc:824] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:42.543782 17347 tablet_bootstrap.cc:492] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Bootstrap starting.
I20260925 02:53:42.544203 17346 tablet_bootstrap.cc:654] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:42.545352 17346 log.cc:824] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:42.547495 17345 tablet_bootstrap.cc:492] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d: No bootstrap required, opened a new log
I20260925 02:53:42.547940 17345 ts_tablet_manager.cc:1397] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260925 02:53:42.547966 17347 tablet_bootstrap.cc:654] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:42.549481 17347 log.cc:824] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:42.549773 17346 tablet_bootstrap.cc:492] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604: No bootstrap required, opened a new log
I20260925 02:53:42.550091 17346 ts_tablet_manager.cc:1397] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 02:53:42.552903 17347 tablet_bootstrap.cc:492] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: No bootstrap required, opened a new log
I20260925 02:53:42.553224 17347 ts_tablet_manager.cc:1397] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 02:53:42.560118 17345 raft_consensus.cc:348] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } } peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } }
I20260925 02:53:42.560621 17345 raft_consensus.cc:374] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:42.560735 17345 raft_consensus.cc:729] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94db12ef415843c0ad414ca2fdd6df2d, State: Initialized, Role: FOLLOWER
I20260925 02:53:42.560995 17346 raft_consensus.cc:348] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } } peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } }
I20260925 02:53:42.561434 17346 raft_consensus.cc:374] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:42.561563 17346 raft_consensus.cc:729] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7729d3bb6f4bf6a16ee6d54cad8604, State: Initialized, Role: FOLLOWER
I20260925 02:53:42.561453 17345 consensus_queue.cc:260] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } } peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } }
I20260925 02:53:42.562021 17347 raft_consensus.cc:348] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } } peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } }
I20260925 02:53:42.562165 17346 consensus_queue.cc:260] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } } peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } }
I20260925 02:53:42.562493 17347 raft_consensus.cc:374] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:42.562609 17347 raft_consensus.cc:729] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e, State: Initialized, Role: FOLLOWER
I20260925 02:53:42.563354 17347 consensus_queue.cc:260] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } } peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } }
I20260925 02:53:42.564390 17345 ts_tablet_manager.cc:1428] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
W20260925 02:53:42.564857 17058 tablet.cc:2369] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:42.565513 17326 heartbeater.cc:503] Master 127.2.172.126:42799 was elected leader, sending a full tablet report...
I20260925 02:53:42.565979 17192 heartbeater.cc:503] Master 127.2.172.126:42799 was elected leader, sending a full tablet report...
I20260925 02:53:42.566709 17346 ts_tablet_manager.cc:1428] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260925 02:53:42.567644 17347 ts_tablet_manager.cc:1428] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
W20260925 02:53:42.615607 17193 tablet.cc:2369] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:53:42.660948 17327 tablet.cc:2369] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:42.664716 17352 raft_consensus.cc:482] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:42.665057 17352 raft_consensus.cc:504] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } } peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } }
I20260925 02:53:42.667182 17352 leader_election.cc:288] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437), 94db12ef415843c0ad414ca2fdd6df2d (127.2.172.65:38737)
I20260925 02:53:42.677136 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.668193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:39445 (local address 127.2.172.65:38737)
0925 02:53:42.668438 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.668446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.668470 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:53:42.668716 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.668722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.668800 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.668908 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:42.669620 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.671057 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.674403 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.675807 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.676004 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.676091 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:42.676685 (+ 594us) server_negotiation.cc:299] Negotiation successful
0925 02:53:42.676841 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:53:42.677824 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.667462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:38737 (local address 127.2.172.66:39445)
0925 02:53:42.668387 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:42.668408 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:42.668554 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:42.668941 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:42.668947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:42.668974 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:42.669449 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:42.669465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.671223 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.671230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:42.674133 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:42.674149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.676268 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.676276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:42.676477 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.676518 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 02:53:42.677377 (+ 859us) client_negotiation.cc:240] Negotiation successful
0925 02:53:42.677540 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":811,"threads_started":1}
I20260925 02:53:42.679039 17012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e181c0876d41c0a7f9d2b8db1cab32" candidate_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94db12ef415843c0ad414ca2fdd6df2d" is_pre_election: true
I20260925 02:53:42.679417 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.667689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:35187 (local address 127.2.172.67:43437)
0925 02:53:42.667919 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.667927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.669096 (+ 1169us) server_negotiation.cc:407] Connection header received
0925 02:53:42.669291 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.669297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.669391 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.669488 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:42.671155 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.672567 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.676216 (+ 3649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.678069 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.678220 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.678302 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:42.678938 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 02:53:42.679111 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 02:53:42.679603 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.667538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:43437 (local address 127.2.172.66:35187)
0925 02:53:42.669001 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:42.669017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:42.669149 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:42.669897 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:42.669902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:42.669932 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:42.670707 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:42.670726 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.672775 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.672780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:42.676033 (+ 3253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:42.676049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.678512 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.678519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:42.678727 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.678760 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:53:42.678888 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 02:53:42.679149 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1323,"thread_start_us":1134,"threads_started":1}
I20260925 02:53:42.679850 17012 raft_consensus.cc:2454] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc7729d3bb6f4bf6a16ee6d54cad8604 in term 0.
I20260925 02:53:42.680718 17079 leader_election.cc:302] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94db12ef415843c0ad414ca2fdd6df2d, fc7729d3bb6f4bf6a16ee6d54cad8604; no voters:
I20260925 02:53:42.680711 17281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e181c0876d41c0a7f9d2b8db1cab32" candidate_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" is_pre_election: true
I20260925 02:53:42.681175 17352 raft_consensus.cc:2781] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:42.681306 17352 raft_consensus.cc:482] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:42.681277 17281 raft_consensus.cc:2454] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc7729d3bb6f4bf6a16ee6d54cad8604 in term 0.
I20260925 02:53:42.681412 17352 raft_consensus.cc:3037] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:42.684928 17352 raft_consensus.cc:504] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } } peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } }
I20260925 02:53:42.686327 17281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e181c0876d41c0a7f9d2b8db1cab32" candidate_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e"
I20260925 02:53:42.686389 17012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e181c0876d41c0a7f9d2b8db1cab32" candidate_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94db12ef415843c0ad414ca2fdd6df2d"
I20260925 02:53:42.686573 17281 raft_consensus.cc:3037] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:42.686625 17012 raft_consensus.cc:3037] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:42.689783 17281 raft_consensus.cc:2454] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc7729d3bb6f4bf6a16ee6d54cad8604 in term 1.
I20260925 02:53:42.690313 17082 leader_election.cc:302] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e, fc7729d3bb6f4bf6a16ee6d54cad8604; no voters:
I20260925 02:53:42.690333 17012 raft_consensus.cc:2454] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc7729d3bb6f4bf6a16ee6d54cad8604 in term 1.
I20260925 02:53:42.690999 17352 leader_election.cc:288] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [CANDIDATE]: Term 1 election: Requested vote from peers 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437), 94db12ef415843c0ad414ca2fdd6df2d (127.2.172.65:38737)
I20260925 02:53:42.691061 17356 raft_consensus.cc:2781] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:42.691336 17356 raft_consensus.cc:686] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 1 LEADER]: Becoming Leader. State: Replica: fc7729d3bb6f4bf6a16ee6d54cad8604, State: Running, Role: LEADER
I20260925 02:53:42.692016 17356 consensus_queue.cc:237] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } } peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } }
I20260925 02:53:42.697870 16868 catalog_manager.cc:5495] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66). New cstate: current_term: 1 leader_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } health_report { overall_health: UNKNOWN } } }
I20260925 02:53:42.759806 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.754568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.2.172.66:39675)
0925 02:53:42.754765 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.754770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.754787 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:53:42.754877 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.754880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.754969 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.755092 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:42.755484 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.756238 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.757572 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.757968 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.758060 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.758560 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:42.759108 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:42.759216 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:42.759341 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 02:53:42.759454 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:53:42.768085 17281 raft_consensus.cc:1261] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 1 FOLLOWER]: Refusing update from remote peer fc7729d3bb6f4bf6a16ee6d54cad8604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:42.768131 17012 raft_consensus.cc:1261] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Refusing update from remote peer fc7729d3bb6f4bf6a16ee6d54cad8604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:42.769045 17356 consensus_queue.cc:1035] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:42.769328 17352 consensus_queue.cc:1035] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:42.781901 17359 mvcc.cc:205] Tried to move back new op lower bound from 7333088554048385024 to 7333088553752444928. Current Snapshot: MvccSnapshot[applied={T|T < 7333088554048385024}]
I20260925 02:53:42.795069 16866 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:42.795362 16866 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:42.799155 16866 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Sending DeleteTablet for 3 replicas of tablet 60e181c0876d41c0a7f9d2b8db1cab32
I20260925 02:53:42.800623 17127 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e181c0876d41c0a7f9d2b8db1cab32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:42 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:52156
I20260925 02:53:42.801095 17261 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e181c0876d41c0a7f9d2b8db1cab32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:42 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:35332
I20260925 02:53:42.801214 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 16929
I20260925 02:53:42.801836 16992 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e181c0876d41c0a7f9d2b8db1cab32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:42 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:55116
I20260925 02:53:42.813006 17369 tablet_replica.cc:330] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604: stopping tablet replica
I20260925 02:53:42.813655 17369 raft_consensus.cc:2229] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:53:42.814498 17369 raft_consensus.cc:2258] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:42.815655 17372 tablet_replica.cc:330] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d: stopping tablet replica
I20260925 02:53:42.816293 17370 tablet_replica.cc:330] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: stopping tablet replica
I20260925 02:53:42.816998 17370 raft_consensus.cc:2229] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:42.817480 17372 raft_consensus.cc:2229] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:42.817632 17369 ts_tablet_manager.cc:1905] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:42.818199 17370 raft_consensus.cc:2258] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:42.818686 17372 raft_consensus.cc:2258] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:42.822393 17370 ts_tablet_manager.cc:1905] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:42.823387 17372 ts_tablet_manager.cc:1905] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:42.825809 17369 ts_tablet_manager.cc:1918] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:42.825995 17369 log.cc:1196] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/60e181c0876d41c0a7f9d2b8db1cab32
Sep 25 02:53:42 dist-test-slave-g1xh krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:53:42.826589 17369 ts_tablet_manager.cc:1939] T 60e181c0876d41c0a7f9d2b8db1cab32 P fc7729d3bb6f4bf6a16ee6d54cad8604: Deleting consensus metadata
I20260925 02:53:42.828997 16853 catalog_manager.cc:4841] TS fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675): tablet 60e181c0876d41c0a7f9d2b8db1cab32 (table test-table [id=0804864e29b748b0a91ac8ab0ef28965]) successfully deleted
I20260925 02:53:42.833491 17372 ts_tablet_manager.cc:1918] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:42.833668 17372 log.cc:1196] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/60e181c0876d41c0a7f9d2b8db1cab32
I20260925 02:53:42.834350 17372 ts_tablet_manager.cc:1939] T 60e181c0876d41c0a7f9d2b8db1cab32 P 94db12ef415843c0ad414ca2fdd6df2d: Deleting consensus metadata
I20260925 02:53:42.834931 17370 ts_tablet_manager.cc:1918] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:42.835155 17370 log.cc:1196] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/60e181c0876d41c0a7f9d2b8db1cab32
I20260925 02:53:42.835748 17370 ts_tablet_manager.cc:1939] T 60e181c0876d41c0a7f9d2b8db1cab32 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Deleting consensus metadata
I20260925 02:53:42.835845 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.802478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.2.172.65:38737)
0925 02:53:42.802711 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.802720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.813395 (+ 10675us) server_negotiation.cc:407] Connection header received
0925 02:53:42.813942 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.813951 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.814057 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.814145 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:42.817390 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.818515 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.821203 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.821843 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.821957 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.827779 (+ 5822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:42.827795 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:42.827817 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:42.827869 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:42.832084 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:42.833049 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:42.833059 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:42.833063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:42.833234 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:42.833709 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:42.833715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:42.833721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:42.834201 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:42.834512 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:42.835142 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 02:53:42.835387 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:53:42.836442 16852 catalog_manager.cc:4841] TS 94db12ef415843c0ad414ca2fdd6df2d (127.2.172.65:38737): tablet 60e181c0876d41c0a7f9d2b8db1cab32 (table test-table [id=0804864e29b748b0a91ac8ab0ef28965]) successfully deleted
I20260925 02:53:42.837500 17052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:42.837508 16855 catalog_manager.cc:4841] TS 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437): tablet 60e181c0876d41c0a7f9d2b8db1cab32 (table test-table [id=0804864e29b748b0a91ac8ab0ef28965]) successfully deleted
I20260925 02:53:42.946789 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 16929
I20260925 02:53:42.973151 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 17063
Sep 25 02:53:42 dist-test-slave-g1xh krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:53:42.992091 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.974070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.2.172.66:39675)
0925 02:53:42.974236 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.974240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.974258 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:53:42.974449 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.974453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.974536 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.974631 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:42.976349 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.977163 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.979130 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:42.979778 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:42.979883 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:42.985602 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:42.985617 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:42.985637 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:42.985681 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:42.989008 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:42.989888 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:42.989895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:42.989900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:42.990051 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:42.990506 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:42.990510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:42.990512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:42.990870 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:42.991151 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:42.991468 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 02:53:42.991727 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:53:42.992937 17187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:43.109642 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 17063
I20260925 02:53:43.130743 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 17198
Sep 25 02:53:43 dist-test-slave-g1xh krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:53:43.150642 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:43.131647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35334 (local address 127.2.172.67:43437)
0925 02:53:43.131849 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:43.131854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:43.131874 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:53:43.132114 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:43.132118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:43.132198 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:43.132291 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:43.133736 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:43.134522 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:43.136263 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:43.137159 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:43.137368 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:43.143891 (+ 6523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:43.143919 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:43.143936 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:43.143993 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:43.147045 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:43.148133 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:43.148142 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:43.148144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:43.148306 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:43.148812 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:43.148816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:43.148819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:43.149275 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:43.149534 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:43.150064 (+ 530us) server_negotiation.cc:299] Negotiation successful
0925 02:53:43.150267 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:53:43.151397 17321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:43.249815 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 17198
I20260925 02:53:43.268218 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 16836
I20260925 02:53:43.269424 16898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:43.381318 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 16836
I20260925 02:53:43.404171 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:42799
--webserver_interface=127.2.172.126
--webserver_port=36787
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:42799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:43.748471 17377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:43.748847 17377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:43.749020 17377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:43.756771 17377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:43.756901 17377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:43.756991 17377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:43.757082 17377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:43.757167 17377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:43.768585 17377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:42799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:42799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=36787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:43.770160 17377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:43.771723 17377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:43.796556 17384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:43.796468 17386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:43.796491 17383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:43.797443 17377 server_base.cc:1034] running on GCE node
I20260925 02:53:43.798388 17377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:43.800279 17377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:43.801561 17377 hybrid_clock.cc:648] HybridClock initialized: now 1790304823801500 us; error 70 us; skew 500 ppm
I20260925 02:53:43.802028 17377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:43 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304823, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:43.810295 17377 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:53:43.812325 17377 webserver.cc:460] Webserver started at http://127.2.172.126:36787/ using document root <none> and password file <none>
I20260925 02:53:43.813166 17377 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:43.813344 17377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:43.819996 17377 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:53:43.823346 17393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:43.825165 17377 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:53:43.825407 17377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "538ffedac61748c4825c95e059559e98"
format_stamp: "Formatted at 2026-09-25 02:53:40 on dist-test-slave-g1xh"
server_key: "474e9e4346243a4cb2899f9e07ac8daf"
server_key_iv: "22e1cac630bbdf3a6564aabfeef222bd"
server_key_version: "encryptionkey@0"
I20260925 02:53:43.826635 17377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:43.848402 17377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:43.853096 17377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:43.853411 17377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:43.873776 17377 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:42799
I20260925 02:53:43.873796 17444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:42799 every 8 connection(s)
I20260925 02:53:43.875424 17377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:43.880777 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 17377
I20260925 02:53:43.881771 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:38737
--local_ip_for_outbound_sockets=127.2.172.65
--tserver_master_addrs=127.2.172.126:42799
--webserver_port=38181
--webserver_interface=127.2.172.65
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:53:43.883692 17445 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:53:43.889112 17445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Bootstrap starting.
I20260925 02:53:43.910580 17445 log.cc:824] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:43.923084 17445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:53:43.923733 17445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Bootstrap complete.
I20260925 02:53:43.933415 17445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } }
I20260925 02:53:43.934268 17445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538ffedac61748c4825c95e059559e98, State: Initialized, Role: FOLLOWER
I20260925 02:53:43.934942 17445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } }
I20260925 02:53:43.935197 17445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:43.935321 17445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:43.935518 17445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:53:43.939680 17445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } }
I20260925 02:53:43.940256 17445 leader_election.cc:302] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 538ffedac61748c4825c95e059559e98; no voters:
I20260925 02:53:43.940990 17445 leader_election.cc:288] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:53:43.941139 17449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:53:43.942672 17449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 2 LEADER]: Becoming Leader. State: Replica: 538ffedac61748c4825c95e059559e98, State: Running, Role: LEADER
I20260925 02:53:43.943280 17449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } }
I20260925 02:53:43.943847 17445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:43.945917 17450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "538ffedac61748c4825c95e059559e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } } }
I20260925 02:53:43.946347 17450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:43.950304 17451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 538ffedac61748c4825c95e059559e98. Latest consensus state: current_term: 2 leader_uuid: "538ffedac61748c4825c95e059559e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } } }
I20260925 02:53:43.950680 17451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:43.952426 17455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:43.961301 17455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:43.962896 17455 catalog_manager.cc:1259] Loaded cluster ID: fe4fa6fe738f496b9ca241bddf2831b5
I20260925 02:53:43.963063 17455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:43.966246 17455 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 02:53:43.978665 17466 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:53:43.984663 17455 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 02:53:43.994822 17455 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:43.996781 17455 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Loaded TSK: 0
I20260925 02:53:43.998466 17455 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:53:44.250406 17447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:44.250699 17447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:44.250854 17447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:44.259104 17447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:44.259231 17447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:44.259450 17447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:53:44.269958 17447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:38737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=38181
--tserver_master_addrs=127.2.172.126:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:44.271369 17447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:44.272794 17447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:44.287932 17472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:44.287902 17473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:44.292663 17475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:44.293599 17447 server_base.cc:1034] running on GCE node
I20260925 02:53:44.294296 17447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:44.295356 17447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:44.296605 17447 hybrid_clock.cc:648] HybridClock initialized: now 1790304824296549 us; error 46 us; skew 500 ppm
I20260925 02:53:44.296965 17447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:44 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:44.300427 17447 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:53:44.302260 17447 webserver.cc:460] Webserver started at http://127.2.172.65:38181/ using document root <none> and password file <none>
I20260925 02:53:44.303104 17447 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:44.303288 17447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:44.309667 17447 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:53:44.312884 17482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:44.314626 17447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:53:44.314845 17447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "94db12ef415843c0ad414ca2fdd6df2d"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-g1xh"
server_key: "1be9fafad0527f00e401bcf0bc844422"
server_key_iv: "c0bd2c670771298c8b92fedad4eb8865"
server_key_version: "encryptionkey@0"
I20260925 02:53:44.316197 17447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:44.344107 17447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:44.349030 17447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:44.349385 17447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:44.350502 17447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:44.352038 17447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:44.352174 17447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:44.352329 17447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:44.352429 17447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:44.389519 17447 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:38737
I20260925 02:53:44.389611 17594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:38737 every 8 connection(s)
I20260925 02:53:44.391116 17447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:44.401134 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 17447
I20260925 02:53:44.401928 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:39675
--local_ip_for_outbound_sockets=127.2.172.66
--tserver_master_addrs=127.2.172.126:42799
--webserver_port=42619
--webserver_interface=127.2.172.66
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:53:44 dist-test-slave-g1xh krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:44.418674 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.393896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:36957 (local address 127.2.172.126:42799)
0925 02:53:44.394576 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:44.394588 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:44.395038 (+ 450us) server_negotiation.cc:407] Connection header received
0925 02:53:44.397208 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:44.397221 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:44.397542 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:44.397831 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:44.399853 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.401600 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.403388 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.404610 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.404823 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.409140 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:44.409156 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:44.409180 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:44.409239 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:44.413239 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:44.414421 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:44.414429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:44.414435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:44.414629 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:44.415230 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:44.415235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:44.415237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:44.415929 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:44.416130 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:44.417133 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0925 02:53:44.417500 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":317,"threads_started":1}
I20260925 02:53:44.418972 17598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.394010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:36957)
0925 02:53:44.394836 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:44.394889 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:44.396722 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:44.397979 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:44.397986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:44.398547 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:44.399414 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.399440 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.401804 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.401809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.403208 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.403223 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.404886 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.404892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.405098 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.405937 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:44.405976 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:44.408768 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:44.413458 (+ 4690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:44.413472 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:44.413492 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:44.414240 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:44.414764 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:44.414772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:44.414779 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:44.415052 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:44.416090 (+ 1038us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:44.416101 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:44.416580 (+ 479us) client_negotiation.cc:769] Sending connection context
0925 02:53:44.416899 (+ 319us) client_negotiation.cc:240] Negotiation successful
0925 02:53:44.417573 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":365,"threads_started":1}
I20260925 02:53:44.422617 17595 heartbeater.cc:347] Connected to a master server at 127.2.172.126:42799
I20260925 02:53:44.423012 17595 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:44.423983 17595 heartbeater.cc:511] Master 127.2.172.126:42799 requested a full tablet report, sending...
I20260925 02:53:44.426950 17410 ts_manager.cc:194] Registered new tserver with Master: 94db12ef415843c0ad414ca2fdd6df2d (127.2.172.65:38737)
I20260925 02:53:44.429484 17410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:36957
W20260925 02:53:44.747789 17599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:44.748073 17599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:44.748167 17599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:44.755457 17599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:44.755648 17599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:44.755779 17599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:53:44.765672 17599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:39675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=42619
--tserver_master_addrs=127.2.172.126:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:44.766947 17599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:44.768333 17599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:44.781924 17607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:44.782083 17604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:44.781914 17605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:44.782802 17599 server_base.cc:1034] running on GCE node
I20260925 02:53:44.783455 17599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:44.784459 17599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:44.785701 17599 hybrid_clock.cc:648] HybridClock initialized: now 1790304824785651 us; error 42 us; skew 500 ppm
I20260925 02:53:44.786052 17599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:44 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:44.789372 17599 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:53:44.790814 17599 webserver.cc:460] Webserver started at http://127.2.172.66:42619/ using document root <none> and password file <none>
I20260925 02:53:44.791486 17599 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:44.791659 17599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:44.797194 17599 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 02:53:44.800046 17614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:44.801403 17599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:53:44.801594 17599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-g1xh"
server_key: "d216ab606a46290afd34d36820f009ed"
server_key_iv: "ee74c953a60988edca05ca84f177487c"
server_key_version: "encryptionkey@0"
I20260925 02:53:44.802567 17599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:44.816440 17599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:44.820957 17599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:44.821249 17599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:44.822264 17599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:44.823650 17599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:44.823736 17599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:44.823815 17599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:44.823870 17599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:44.859714 17599 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:39675
I20260925 02:53:44.859838 17726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:39675 every 8 connection(s)
I20260925 02:53:44.861296 17599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:44.862677 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 17599
I20260925 02:53:44.863447 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:43437
--local_ip_for_outbound_sockets=127.2.172.67
--tserver_master_addrs=127.2.172.126:42799
--webserver_port=35813
--webserver_interface=127.2.172.67
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:53:44 dist-test-slave-g1xh krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:44.885385 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.864387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:36659 (local address 127.2.172.126:42799)
0925 02:53:44.864688 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:44.864697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:44.865937 (+ 1240us) server_negotiation.cc:407] Connection header received
0925 02:53:44.869023 (+ 3086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:44.869032 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:44.869137 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:44.869235 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:44.871448 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.872507 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.874640 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.875116 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.875264 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.878898 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:44.878903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:44.878906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:44.878949 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:44.881987 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:44.882785 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:44.882792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:44.882795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:44.882952 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:44.883427 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:44.883432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:44.883435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:44.883812 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:44.883982 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:44.884740 (+ 758us) server_negotiation.cc:299] Negotiation successful
0925 02:53:44.885000 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260925 02:53:44.886593 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.864892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:36659)
0925 02:53:44.865734 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:44.865789 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:44.868616 (+ 2827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:44.869535 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:44.869550 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:44.870365 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:44.871230 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.871259 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.872687 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.872693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.874456 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.874472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.875380 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.875388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.875687 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.876409 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:44.876433 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:44.878647 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:44.882144 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:44.882154 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:44.882173 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:44.882648 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:44.883075 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:44.883080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:44.883084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:44.883295 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:44.883951 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:44.883960 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:44.884340 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 02:53:44.884638 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 02:53:44.885216 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":315,"threads_started":1}
I20260925 02:53:44.888193 17727 heartbeater.cc:347] Connected to a master server at 127.2.172.126:42799
I20260925 02:53:44.888522 17727 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:44.889256 17727 heartbeater.cc:511] Master 127.2.172.126:42799 requested a full tablet report, sending...
I20260925 02:53:44.890754 17410 ts_manager.cc:194] Registered new tserver with Master: fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675)
I20260925 02:53:44.891736 17410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:36659
W20260925 02:53:45.214998 17730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:45.215283 17730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:45.215380 17730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:45.222929 17730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:45.223055 17730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:45.223170 17730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:53:45.233517 17730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:43437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=35813
--tserver_master_addrs=127.2.172.126:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:45.234838 17730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:45.236246 17730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:45.251516 17738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:45.251638 17736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:45.251605 17735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:45.252364 17730 server_base.cc:1034] running on GCE node
I20260925 02:53:45.253098 17730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:45.254242 17730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:45.255481 17730 hybrid_clock.cc:648] HybridClock initialized: now 1790304825255436 us; error 55 us; skew 500 ppm
I20260925 02:53:45.255846 17730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:45 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:45.259111 17730 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:53:45.260573 17730 webserver.cc:460] Webserver started at http://127.2.172.67:35813/ using document root <none> and password file <none>
I20260925 02:53:45.261266 17730 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:45.261382 17730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:45.266908 17730 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:53:45.269987 17745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:45.271651 17730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:53:45.271862 17730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e"
format_stamp: "Formatted at 2026-09-25 02:53:42 on dist-test-slave-g1xh"
server_key: "693c1955b114b37c5252973dcd95c1c4"
server_key_iv: "d7393604a2a6e9224860fcde41a03e1d"
server_key_version: "encryptionkey@0"
I20260925 02:53:45.272897 17730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:45.293283 17730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:45.297747 17730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:45.298049 17730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:45.299073 17730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:45.300437 17730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:45.300526 17730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:45.300694 17730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:45.300791 17730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:45.335906 17730 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:43437
I20260925 02:53:45.335959 17857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:43437 every 8 connection(s)
I20260925 02:53:45.337405 17730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:45.345364 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 17730
Sep 25 02:53:45 dist-test-slave-g1xh krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:53:45.358709 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.340049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:33869 (local address 127.2.172.126:42799)
0925 02:53:45.340234 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:45.340239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:45.341319 (+ 1080us) server_negotiation.cc:407] Connection header received
0925 02:53:45.343112 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:45.343118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:45.343206 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:45.343323 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:45.345051 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.345843 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.347953 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.348669 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.348817 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.352420 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:45.352430 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:45.352434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:45.352465 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:45.355300 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:45.356249 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:45.356255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:45.356258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:45.356419 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:45.356952 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:45.356960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:45.356964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:45.357358 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:45.357530 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:45.358200 (+ 670us) server_negotiation.cc:299] Negotiation successful
0925 02:53:45.358424 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:53:45.359956 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.340494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:33869)
0925 02:53:45.341168 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:45.341204 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:45.342838 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:45.343618 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:45.343627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:45.344237 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:45.344775 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:45.344791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.346046 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.346050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:45.347802 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:45.347815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.348877 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.348920 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:45.349164 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.349878 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:45.349905 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:45.352171 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:45.355447 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:45.355456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:45.355474 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:45.356112 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:45.356572 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:45.356576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:45.356580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:45.356824 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:45.357490 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:45.357495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:45.357820 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 02:53:45.358057 (+ 237us) client_negotiation.cc:240] Negotiation successful
0925 02:53:45.358512 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":255,"threads_started":1}
I20260925 02:53:45.361164 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.346387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45232 (local address 127.2.172.126:42799)
0925 02:53:45.347254 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:45.347258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:45.347271 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:53:45.347310 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:45.347313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:45.347364 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:45.347477 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:45.349430 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.350213 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.351740 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.352168 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.352300 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.355237 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:45.355243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:45.355246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:45.355300 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:45.358270 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:45.359129 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:45.359138 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:45.359140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:45.359247 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:45.359687 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:45.359691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:45.359694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:45.359982 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:45.360260 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:45.360652 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 02:53:45.360832 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"spinlock_wait_cycles":5504,"thread_start_us":620,"threads_started":1}
I20260925 02:53:45.361627 17858 heartbeater.cc:347] Connected to a master server at 127.2.172.126:42799
I20260925 02:53:45.361959 17858 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:45.362757 17858 heartbeater.cc:511] Master 127.2.172.126:42799 requested a full tablet report, sending...
I20260925 02:53:45.364239 17410 ts_manager.cc:194] Registered new tserver with Master: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437)
I20260925 02:53:45.365116 17410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:33869
I20260925 02:53:45.376005 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:53:45.383122 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.376986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.2.172.126:42799)
0925 02:53:45.377201 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:45.377210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:45.377642 (+ 432us) server_negotiation.cc:407] Connection header received
0925 02:53:45.377958 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:45.377962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:45.378102 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:45.378267 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:45.378704 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.379443 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.380899 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.381305 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.381451 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.381975 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:45.382507 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:45.382679 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:45.382772 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 02:53:45.382855 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:53:45.384080 17410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:53:45.392386 17410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:45.427435 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.404744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:43437 (local address 127.0.0.1:35336)
0925 02:53:45.418187 (+ 13443us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:45.418522 (+ 335us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:45.418801 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:45.419985 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:45.419993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:45.420034 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:45.420983 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:45.420997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.422551 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.422555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:45.424827 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:45.424846 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.426351 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.426358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:45.426551 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.426604 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 02:53:45.426762 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 02:53:45.426971 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13233,"mutex_wait_us":114,"spinlock_wait_cycles":2816,"thread_start_us":13082,"threads_started":1}
I20260925 02:53:45.428622 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.404966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.2.172.67:43437)
0925 02:53:45.416842 (+ 11876us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:45.416856 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:45.418616 (+ 1760us) server_negotiation.cc:407] Connection header received
0925 02:53:45.419094 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:45.419103 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:45.419399 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:45.419685 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:45.421183 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.422347 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.424998 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.426122 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.426269 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.426521 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:45.428099 (+ 1578us) server_negotiation.cc:299] Negotiation successful
0925 02:53:45.428279 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11588,"thread_start_us":11897,"threads_started":1}
I20260925 02:53:45.428848 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.407410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39675 (local address 127.0.0.1:52184)
0925 02:53:45.418988 (+ 11578us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:45.419007 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:45.419155 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:45.420027 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:45.420031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:45.420053 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:45.420737 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:45.420752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.422871 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.422875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:45.425681 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:45.425690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.427814 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.427820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:45.428033 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.428065 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 02:53:45.428154 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:53:45.428288 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11444,"mutex_wait_us":270,"thread_start_us":11275,"threads_started":1}
I20260925 02:53:45.429162 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.408053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.2.172.66:39675)
0925 02:53:45.418753 (+ 10700us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:45.418762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:45.419288 (+ 526us) server_negotiation.cc:407] Connection header received
0925 02:53:45.419421 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:45.419429 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:45.419687 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:45.419885 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:45.420970 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.422695 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.425866 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.427298 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.427434 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.427630 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:45.428629 (+ 999us) server_negotiation.cc:299] Negotiation successful
0925 02:53:45.428819 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10538,"thread_start_us":10382,"threads_started":1}
I20260925 02:53:45.429296 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.404728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.2.172.65:38737)
0925 02:53:45.417615 (+ 12887us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:45.417625 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:45.418444 (+ 819us) server_negotiation.cc:407] Connection header received
0925 02:53:45.419299 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:45.419306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:45.419530 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:45.419691 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:45.421086 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.422709 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.425881 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.427302 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.427468 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.427735 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:45.428761 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0925 02:53:45.428952 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12715,"thread_start_us":12574,"threads_started":1}
I20260925 02:53:45.429342 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.404436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:38737 (local address 127.0.0.1:55142)
0925 02:53:45.418187 (+ 13751us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:45.418275 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:45.419108 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:45.419985 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:45.419993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:45.420041 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:45.420861 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:45.420876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.422868 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.422873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:45.425674 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:45.425690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.427887 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.427893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:45.428081 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.428112 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:53:45.428194 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:53:45.428329 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13432,"mutex_wait_us":204,"spinlock_wait_cycles":8960,"thread_start_us":13268,"threads_started":1}
I20260925 02:53:45.432682 17530 tablet_service.cc:1467] Processing CreateTablet for tablet 78068476956c40d7b8dcaaeda8445ce7 (DEFAULT_TABLE table=test-table [id=a74caaf0eb644e3ea96efb0070cf15e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:45.432732 17662 tablet_service.cc:1467] Processing CreateTablet for tablet 78068476956c40d7b8dcaaeda8445ce7 (DEFAULT_TABLE table=test-table [id=a74caaf0eb644e3ea96efb0070cf15e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:45.432732 17793 tablet_service.cc:1467] Processing CreateTablet for tablet 78068476956c40d7b8dcaaeda8445ce7 (DEFAULT_TABLE table=test-table [id=a74caaf0eb644e3ea96efb0070cf15e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:45.434022 17793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78068476956c40d7b8dcaaeda8445ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:45.434023 17662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78068476956c40d7b8dcaaeda8445ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:45.434022 17530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78068476956c40d7b8dcaaeda8445ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:45.434072 17595 heartbeater.cc:503] Master 127.2.172.126:42799 was elected leader, sending a full tablet report...
I20260925 02:53:45.446339 17874 tablet_bootstrap.cc:492] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: Bootstrap starting.
I20260925 02:53:45.449519 17875 tablet_bootstrap.cc:492] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604: Bootstrap starting.
I20260925 02:53:45.450155 17874 tablet_bootstrap.cc:654] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:45.451519 17874 log.cc:824] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:45.451812 17876 tablet_bootstrap.cc:492] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Bootstrap starting.
I20260925 02:53:45.453501 17875 tablet_bootstrap.cc:654] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:45.454839 17875 log.cc:824] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:45.455120 17876 tablet_bootstrap.cc:654] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:45.455462 17874 tablet_bootstrap.cc:492] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: No bootstrap required, opened a new log
I20260925 02:53:45.455847 17874 ts_tablet_manager.cc:1397] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260925 02:53:45.456528 17876 log.cc:824] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:45.457988 17875 tablet_bootstrap.cc:492] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604: No bootstrap required, opened a new log
I20260925 02:53:45.458313 17875 ts_tablet_manager.cc:1397] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 02:53:45.460775 17876 tablet_bootstrap.cc:492] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: No bootstrap required, opened a new log
I20260925 02:53:45.461035 17876 ts_tablet_manager.cc:1397] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 02:53:45.466522 17874 raft_consensus.cc:348] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:45.466998 17874 raft_consensus.cc:374] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:45.467144 17874 raft_consensus.cc:729] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94db12ef415843c0ad414ca2fdd6df2d, State: Initialized, Role: FOLLOWER
I20260925 02:53:45.467805 17874 consensus_queue.cc:260] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:45.468071 17876 raft_consensus.cc:348] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:45.468547 17876 raft_consensus.cc:374] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:45.468698 17876 raft_consensus.cc:729] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e, State: Initialized, Role: FOLLOWER
I20260925 02:53:45.468793 17875 raft_consensus.cc:348] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:45.469376 17875 raft_consensus.cc:374] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:45.469455 17876 consensus_queue.cc:260] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:45.469650 17875 raft_consensus.cc:729] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7729d3bb6f4bf6a16ee6d54cad8604, State: Initialized, Role: FOLLOWER
I20260925 02:53:45.470516 17875 consensus_queue.cc:260] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:45.471225 17874 ts_tablet_manager.cc:1428] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260925 02:53:45.472478 17727 heartbeater.cc:503] Master 127.2.172.126:42799 was elected leader, sending a full tablet report...
I20260925 02:53:45.472985 17875 ts_tablet_manager.cc:1428] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 02:53:45.474632 17858 heartbeater.cc:503] Master 127.2.172.126:42799 was elected leader, sending a full tablet report...
I20260925 02:53:45.475244 17876 ts_tablet_manager.cc:1428] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260925 02:53:45.589318 17859 tablet.cc:2369] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:53:45.613448 17728 tablet.cc:2369] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:45.639187 17881 raft_consensus.cc:482] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:45.639605 17881 raft_consensus.cc:504] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:45.641389 17881 leader_election.cc:288] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94db12ef415843c0ad414ca2fdd6df2d (127.2.172.65:38737), fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675)
W20260925 02:53:45.643639 17596 tablet.cc:2369] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:45.647627 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.641805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:37895 (local address 127.2.172.65:38737)
0925 02:53:45.641993 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:45.641999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:45.642076 (+ 77us) server_negotiation.cc:407] Connection header received
0925 02:53:45.642301 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:45.642309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:45.642400 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:45.642520 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:45.642941 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.644103 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.645953 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.646712 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.646877 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.646950 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:45.647213 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 02:53:45.647337 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:53:45.647629 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.641611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:38737 (local address 127.2.172.67:37895)
0925 02:53:45.641967 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:45.641989 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:45.642155 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:45.642580 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:45.642585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:45.642624 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:45.642818 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:45.642827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.644241 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.644246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:45.645805 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:45.645817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.646866 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.646872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:45.647059 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.647082 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:53:45.647187 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 02:53:45.647314 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260925 02:53:45.648392 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.642014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:54213 (local address 127.2.172.66:39675)
0925 02:53:45.642243 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:45.642252 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:45.642830 (+ 578us) server_negotiation.cc:407] Connection header received
0925 02:53:45.643031 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:45.643035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:45.643093 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:45.643160 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:45.643763 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.644843 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.646518 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.647275 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.647446 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.647531 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:45.647967 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 02:53:45.648123 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:53:45.648671 17550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78068476956c40d7b8dcaaeda8445ce7" candidate_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94db12ef415843c0ad414ca2fdd6df2d" is_pre_election: true
I20260925 02:53:45.649197 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.641932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39675 (local address 127.2.172.67:54213)
0925 02:53:45.642729 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:45.642742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:45.642875 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:45.643214 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:45.643220 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:45.643247 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:45.643596 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:45.643615 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.644990 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.644995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:45.646351 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:45.646362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.647479 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.647489 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:45.647670 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.647815 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 02:53:45.648619 (+ 804us) client_negotiation.cc:240] Negotiation successful
0925 02:53:45.648788 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":5,"thread_start_us":465,"threads_started":1}
I20260925 02:53:45.649299 17550 raft_consensus.cc:2454] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bb7ccf8300d4b6d84ac3a7c8aa8f06e in term 0.
I20260925 02:53:45.650043 17746 leader_election.cc:302] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e, 94db12ef415843c0ad414ca2fdd6df2d; no voters:
I20260925 02:53:45.650483 17881 raft_consensus.cc:2781] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:45.650482 17682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78068476956c40d7b8dcaaeda8445ce7" candidate_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" is_pre_election: true
I20260925 02:53:45.650660 17881 raft_consensus.cc:482] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:45.650769 17881 raft_consensus.cc:3037] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:45.651049 17682 raft_consensus.cc:2454] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bb7ccf8300d4b6d84ac3a7c8aa8f06e in term 0.
I20260925 02:53:45.653393 17881 raft_consensus.cc:504] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:45.653954 17881 leader_election.cc:288] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [CANDIDATE]: Term 1 election: Requested vote from peers 94db12ef415843c0ad414ca2fdd6df2d (127.2.172.65:38737), fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675)
I20260925 02:53:45.654480 17682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78068476956c40d7b8dcaaeda8445ce7" candidate_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604"
I20260925 02:53:45.654518 17550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78068476956c40d7b8dcaaeda8445ce7" candidate_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94db12ef415843c0ad414ca2fdd6df2d"
I20260925 02:53:45.654700 17682 raft_consensus.cc:3037] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:45.654717 17550 raft_consensus.cc:3037] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:45.657416 17550 raft_consensus.cc:2454] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bb7ccf8300d4b6d84ac3a7c8aa8f06e in term 1.
I20260925 02:53:45.657428 17682 raft_consensus.cc:2454] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bb7ccf8300d4b6d84ac3a7c8aa8f06e in term 1.
I20260925 02:53:45.657845 17746 leader_election.cc:302] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e, 94db12ef415843c0ad414ca2fdd6df2d; no voters:
I20260925 02:53:45.658151 17881 raft_consensus.cc:2781] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:45.658619 17881 raft_consensus.cc:686] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 1 LEADER]: Becoming Leader. State: Replica: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e, State: Running, Role: LEADER
I20260925 02:53:45.659085 17881 consensus_queue.cc:237] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:45.663975 17409 catalog_manager.cc:5495] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67). New cstate: current_term: 1 leader_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } health_report { overall_health: UNKNOWN } } }
I20260925 02:53:45.726477 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.720765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.2.172.67:43437)
0925 02:53:45.721013 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:45.721020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:45.721041 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:53:45.721277 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:45.721281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:45.721462 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:45.721637 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:45.722005 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.722730 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.724242 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.724666 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.724808 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.725319 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:45.725826 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:45.726001 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:45.726061 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 02:53:45.726144 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:53:45.735029 17682 raft_consensus.cc:1261] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 1 FOLLOWER]: Refusing update from remote peer 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:45.735028 17550 raft_consensus.cc:1261] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Refusing update from remote peer 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:45.736135 17884 consensus_queue.cc:1035] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:45.736562 17881 consensus_queue.cc:1035] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:45.764314 17410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:45.764603 17410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:45.768085 17410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Sending DeleteTablet for 3 replicas of tablet 78068476956c40d7b8dcaaeda8445ce7
I20260925 02:53:45.769208 17793 tablet_service.cc:1514] Processing DeleteTablet for tablet 78068476956c40d7b8dcaaeda8445ce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:45 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:35336
I20260925 02:53:45.769409 17662 tablet_service.cc:1514] Processing DeleteTablet for tablet 78068476956c40d7b8dcaaeda8445ce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:45 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:52184
I20260925 02:53:45.769984 17530 tablet_service.cc:1514] Processing DeleteTablet for tablet 78068476956c40d7b8dcaaeda8445ce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:45 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:55142
I20260925 02:53:45.770116 17897 tablet_replica.cc:330] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: stopping tablet replica
I20260925 02:53:45.770542 17897 raft_consensus.cc:2229] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:53:45.770875 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 17447
I20260925 02:53:45.771260 17897 raft_consensus.cc:2258] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:45.773401 17897 ts_tablet_manager.cc:1905] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:45.782577 17897 ts_tablet_manager.cc:1918] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:45.782851 17897 log.cc:1196] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/78068476956c40d7b8dcaaeda8445ce7
I20260925 02:53:45.783684 17897 ts_tablet_manager.cc:1939] T 78068476956c40d7b8dcaaeda8445ce7 P 1bb7ccf8300d4b6d84ac3a7c8aa8f06e: Deleting consensus metadata
I20260925 02:53:45.785975 17397 catalog_manager.cc:4841] TS 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437): tablet 78068476956c40d7b8dcaaeda8445ce7 (table test-table [id=a74caaf0eb644e3ea96efb0070cf15e4]) successfully deleted
I20260925 02:53:45.789239 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.772206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.2.172.65:38737)
0925 02:53:45.772373 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:45.772391 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:45.772410 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:53:45.772466 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:45.772472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:45.772554 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:45.772659 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:45.774485 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.775691 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.777504 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.778207 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.778328 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.781750 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:45.781767 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:45.781786 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:45.781834 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:45.785838 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:45.786688 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:45.786702 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:45.786708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:45.786943 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:45.787329 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:45.787336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:45.787339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:45.787967 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:45.788254 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:45.788575 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 02:53:45.788798 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:53:45.790396 17590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:45.791039 17898 tablet_replica.cc:330] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604: stopping tablet replica
I20260925 02:53:45.791638 17898 raft_consensus.cc:2229] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:45.792331 17898 raft_consensus.cc:2258] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:45.794643 17898 ts_tablet_manager.cc:1905] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:45.802088 17898 ts_tablet_manager.cc:1918] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:45.802273 17898 log.cc:1196] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/78068476956c40d7b8dcaaeda8445ce7
I20260925 02:53:45.802838 17898 ts_tablet_manager.cc:1939] T 78068476956c40d7b8dcaaeda8445ce7 P fc7729d3bb6f4bf6a16ee6d54cad8604: Deleting consensus metadata
I20260925 02:53:45.804934 17395 catalog_manager.cc:4841] TS fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675): tablet 78068476956c40d7b8dcaaeda8445ce7 (table test-table [id=a74caaf0eb644e3ea96efb0070cf15e4]) successfully deleted
I20260925 02:53:45.893442 17899 tablet_replica.cc:330] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: stopping tablet replica
I20260925 02:53:45.893975 17899 raft_consensus.cc:2229] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:45.894193 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 17447
W20260925 02:53:45.920641 17394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.172.65:38737 (error 108)
I20260925 02:53:45.920975 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 17599
I20260925 02:53:45.922819 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.922312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:38737 (local address 127.0.0.1:55166)
0925 02:53:45.922518 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:45.922638 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 02:53:45.923087 17394 catalog_manager.cc:4584] TS 94db12ef415843c0ad414ca2fdd6df2d (127.2.172.65:38737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78068476956c40d7b8dcaaeda8445ce7: Network error: Client connection negotiation failed: client connection to 127.2.172.65:38737: connect: Connection refused (error 111)
I20260925 02:53:45.938035 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.922292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.2.172.66:39675)
0925 02:53:45.922466 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:45.922472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:45.922491 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:53:45.922559 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:45.922564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:45.922645 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:45.922754 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:45.924613 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.925403 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.927325 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:45.927878 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:45.927998 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:45.931463 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:45.931479 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:45.931495 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:45.931537 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:45.934720 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:45.935709 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:45.935721 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:45.935723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:45.935899 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:45.936378 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:45.936382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:45.936384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:45.936810 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:45.937056 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:45.937496 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 02:53:45.937673 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:53:45.938723 17722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:45.976897 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.976375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:38737 (local address 127.0.0.1:55176)
0925 02:53:45.976605 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:45.976715 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 02:53:45.978190 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:45.977733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:38737 (local address 127.0.0.1:55192)
0925 02:53:45.977920 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:45.978024 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 02:53:46.031504 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 17599
I20260925 02:53:46.039325 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:46.038668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:38737 (local address 127.0.0.1:55200)
0925 02:53:46.038951 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:46.039082 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 02:53:46.040601 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:46.040203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:38737 (local address 127.0.0.1:55216)
0925 02:53:46.040374 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:46.040465 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 02:53:46.051527 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 17730
I20260925 02:53:46.066202 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:46.052502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.2.172.67:43437)
0925 02:53:46.052699 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:46.052705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:46.052725 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:53:46.052799 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:46.052802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:46.052874 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:46.052998 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:46.054411 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:46.055235 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:46.056816 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:46.057281 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:46.057418 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:46.060129 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:46.060141 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:46.060156 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:46.060196 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:46.063263 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:46.064123 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:46.064130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:46.064132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:46.064283 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:46.064665 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:46.064669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:46.064671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:46.065007 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:46.065311 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:46.065620 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 02:53:46.065845 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:53:46.067396 17853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:46.121927 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:46.121369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:38737 (local address 127.0.0.1:55218)
0925 02:53:46.121632 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:46.121727 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260925 02:53:46.123164 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:46.122739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:38737 (local address 127.0.0.1:55222)
0925 02:53:46.122973 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:46.123047 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 02:53:46.191692 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 17730
I20260925 02:53:46.210547 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 17377
I20260925 02:53:46.211740 17440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:53:46.331604 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:46.330699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:38737 (local address 127.0.0.1:55234)
0925 02:53:46.331183 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:46.331317 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325}
I20260925 02:53:46.331626 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 17377
I20260925 02:53:46.354751 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:42799
--webserver_interface=127.2.172.126
--webserver_port=36787
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:42799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:46.691380 17905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:46.691883 17905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:46.692072 17905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:46.699514 17905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:46.699695 17905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:46.699808 17905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:46.699910 17905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:46.699980 17905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:46.709883 17905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:42799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:42799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=36787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:46.711273 17905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:46.712601 17905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:46.724753 17910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:46.724721 17911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:46.725133 17913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:46.725662 17905 server_base.cc:1034] running on GCE node
I20260925 02:53:46.726491 17905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:46.728018 17905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:46.729292 17905 hybrid_clock.cc:648] HybridClock initialized: now 1790304826729199 us; error 67 us; skew 500 ppm
I20260925 02:53:46.729764 17905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:46 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:46.733283 17905 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:53:46.734948 17905 webserver.cc:460] Webserver started at http://127.2.172.126:36787/ using document root <none> and password file <none>
I20260925 02:53:46.735739 17905 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:46.735869 17905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:46.741657 17905 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 02:53:46.745079 17920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:46.746510 17905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:53:46.746662 17905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "538ffedac61748c4825c95e059559e98"
format_stamp: "Formatted at 2026-09-25 02:53:40 on dist-test-slave-g1xh"
server_key: "474e9e4346243a4cb2899f9e07ac8daf"
server_key_iv: "22e1cac630bbdf3a6564aabfeef222bd"
server_key_version: "encryptionkey@0"
I20260925 02:53:46.747733 17905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:46.770972 17905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:46.775199 17905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:46.775434 17905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:46.795181 17905 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:42799
I20260925 02:53:46.795199 17971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:42799 every 8 connection(s)
I20260925 02:53:46.796828 17905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:46.801013 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 17905
I20260925 02:53:46.801913 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:38737
--local_ip_for_outbound_sockets=127.2.172.65
--tserver_master_addrs=127.2.172.126:42799
--webserver_port=38181
--webserver_interface=127.2.172.65
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:53:46.804566 17972 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:53:46.807921 17972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Bootstrap starting.
I20260925 02:53:46.829094 17972 log.cc:824] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:46.845204 17972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:53:46.845801 17972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98: Bootstrap complete.
I20260925 02:53:46.856065 17972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } }
I20260925 02:53:46.857065 17972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538ffedac61748c4825c95e059559e98, State: Initialized, Role: FOLLOWER
I20260925 02:53:46.857713 17972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } }
I20260925 02:53:46.857963 17972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:46.858103 17972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:46.858289 17972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:53:46.862481 17972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } }
I20260925 02:53:46.863036 17972 leader_election.cc:302] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 538ffedac61748c4825c95e059559e98; no voters:
I20260925 02:53:46.863819 17972 leader_election.cc:288] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:53:46.863982 17976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:53:46.865191 17976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [term 3 LEADER]: Becoming Leader. State: Replica: 538ffedac61748c4825c95e059559e98, State: Running, Role: LEADER
I20260925 02:53:46.865803 17976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } }
I20260925 02:53:46.866132 17972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:46.868517 17977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "538ffedac61748c4825c95e059559e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } } }
I20260925 02:53:46.868961 17977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:46.873796 17978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 538ffedac61748c4825c95e059559e98. Latest consensus state: current_term: 3 leader_uuid: "538ffedac61748c4825c95e059559e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538ffedac61748c4825c95e059559e98" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42799 } } }
I20260925 02:53:46.874723 17978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538ffedac61748c4825c95e059559e98 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:46.876959 17982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:46.887707 17982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:46.889973 17982 catalog_manager.cc:1259] Loaded cluster ID: fe4fa6fe738f496b9ca241bddf2831b5
I20260925 02:53:46.890203 17982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 02:53:46.893168 17982 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5a44b9d9ad google::LogMessage::Fail() at ??:0
@ 0x7f5a44ba088a google::LogMessage::SendToLog() at ??:0
@ 0x7f5a44b9d4de google::LogMessage::Flush() at ??:0
I20260925 02:53:46.902631 17993 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f5a44b9ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5a5a5d0512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5a5a5cd976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5a5a655610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5a5a655439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5a477e9345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5a477ec91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5a477ec799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5a477c7eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5a4aaa16db start_thread at ??:0
@ 0x7f5a43a3761f clone at ??:0
W20260925 02:53:47.235157 17974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:47.235472 17974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:47.235652 17974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:47.243268 17974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:47.243400 17974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:47.243527 17974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:53:47.255249 17974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:38737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=38181
--tserver_master_addrs=127.2.172.126:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:47.256641 17974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:47.258061 17974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:47.271893 18016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:47.271986 18013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:47.271966 18014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:47.272852 17974 server_base.cc:1034] running on GCE node
I20260925 02:53:47.273587 17974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:47.274684 17974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:47.275910 17974 hybrid_clock.cc:648] HybridClock initialized: now 1790304827275862 us; error 59 us; skew 500 ppm
I20260925 02:53:47.276269 17974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:47 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304827, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:47.279727 17974 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:53:47.281386 17974 webserver.cc:460] Webserver started at http://127.2.172.65:38181/ using document root <none> and password file <none>
I20260925 02:53:47.282140 17974 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:47.282253 17974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:47.288192 17974 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:53:47.291350 18023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:47.292985 17974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:53:47.293169 17974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "94db12ef415843c0ad414ca2fdd6df2d"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-g1xh"
server_key: "1be9fafad0527f00e401bcf0bc844422"
server_key_iv: "c0bd2c670771298c8b92fedad4eb8865"
server_key_version: "encryptionkey@0"
I20260925 02:53:47.294164 17974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:47.312172 17974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:47.316767 17974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:47.317090 17974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:47.318173 17974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:47.320111 18030 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 02:53:47.326081 17974 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:53:47.326256 17974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260925 02:53:47.326458 17974 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:53:47.329859 17974 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:53:47.329974 17974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.002s
I20260925 02:53:47.330439 18030 tablet_bootstrap.cc:492] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: Bootstrap starting.
I20260925 02:53:47.361243 18030 log.cc:824] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:47.375432 18030 tablet_bootstrap.cc:492] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:53:47.376111 18030 tablet_bootstrap.cc:492] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: Bootstrap complete.
I20260925 02:53:47.377460 17974 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:38737
I20260925 02:53:47.377508 18137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:38737 every 8 connection(s)
I20260925 02:53:47.377692 18030 ts_tablet_manager.cc:1397] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: Time spent bootstrapping tablet: real 0.048s user 0.040s sys 0.004s
I20260925 02:53:47.378973 17974 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:47.379751 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 17974
I20260925 02:53:47.380534 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:39675
--local_ip_for_outbound_sockets=127.2.172.66
--tserver_master_addrs=127.2.172.126:42799
--webserver_port=42619
--webserver_interface=127.2.172.66
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:53:47.387929 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:47.385430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:47521)
0925 02:53:47.386436 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:47.386737 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":434,"threads_started":1}
W20260925 02:53:47.388695 18138 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:42799 (0 consecutive failures): Network error: Failed to ping master at 127.2.172.126:42799: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
I20260925 02:53:47.389799 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:47.389386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:36323)
0925 02:53:47.389589 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:47.389674 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 02:53:47.388984 18030 raft_consensus.cc:348] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:47.390583 18030 raft_consensus.cc:729] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94db12ef415843c0ad414ca2fdd6df2d, State: Initialized, Role: FOLLOWER
I20260925 02:53:47.391381 18030 consensus_queue.cc:260] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:47.391686 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:47.390319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:44485)
0925 02:53:47.391131 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:47.391299 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":367,"threads_started":1}
W20260925 02:53:47.392050 18138 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:53:47.393374 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:47.393024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:57603)
0925 02:53:47.393179 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:47.393259 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 02:53:47.393607 18030 ts_tablet_manager.cc:1428] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d: Time spent starting tablet: real 0.016s user 0.009s sys 0.008s
I20260925 02:53:47.394708 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:47.394236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:38265)
0925 02:53:47.394425 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:47.394530 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:53:47.750274 18143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:47.750593 18143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:47.750752 18143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:47.758517 18143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:47.758656 18143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:47.758857 18143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:53:47.769229 18143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:39675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=42619
--tserver_master_addrs=127.2.172.126:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:47.770622 18143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:47.772078 18143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:47.786279 18148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:47.786422 18151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:47.786428 18149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:47.786911 18143 server_base.cc:1034] running on GCE node
I20260925 02:53:47.787742 18143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:47.788803 18143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:47.790026 18143 hybrid_clock.cc:648] HybridClock initialized: now 1790304827789951 us; error 59 us; skew 500 ppm
I20260925 02:53:47.790375 18143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:47 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304827, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:47.793735 18143 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:53:47.795367 18143 webserver.cc:460] Webserver started at http://127.2.172.66:42619/ using document root <none> and password file <none>
I20260925 02:53:47.796182 18143 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:47.796324 18143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:47.802150 18143 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:53:47.805310 18158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:47.806779 18143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:53:47.806977 18143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-g1xh"
server_key: "d216ab606a46290afd34d36820f009ed"
server_key_iv: "ee74c953a60988edca05ca84f177487c"
server_key_version: "encryptionkey@0"
I20260925 02:53:47.808074 18143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:47.821547 18143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:47.825881 18143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:47.826268 18143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:47.827289 18143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:47.828665 18143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:47.828791 18143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:47.828936 18143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:47.829033 18143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:47.864188 18143 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:39675
I20260925 02:53:47.864275 18270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:39675 every 8 connection(s)
I20260925 02:53:47.865664 18143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:47.870324 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:47.868518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:49751)
0925 02:53:47.869228 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:47.869485 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":272,"threads_started":1}
W20260925 02:53:47.871043 18271 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:42799 (0 consecutive failures): Network error: Failed to ping master at 127.2.172.126:42799: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
I20260925 02:53:47.871917 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:47.871611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:58129)
0925 02:53:47.871740 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:47.871806 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 02:53:47.872725 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:47.872441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:36295)
0925 02:53:47.872557 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:47.872624 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 02:53:47.872972 18271 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:53:47.874233 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 18143
I20260925 02:53:47.874819 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:43437
--local_ip_for_outbound_sockets=127.2.172.67
--tserver_master_addrs=127.2.172.126:42799
--webserver_port=35813
--webserver_interface=127.2.172.67
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:48.207297 18274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:48.207687 18274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:48.207857 18274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:48.215451 18274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:48.215660 18274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:48.215857 18274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:53:48.225979 18274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:43437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=35813
--tserver_master_addrs=127.2.172.126:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:53:48.227517 18274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:48.229058 18274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:48.246590 18279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:48.246547 18282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:48.246624 18280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:48.247941 18274 server_base.cc:1034] running on GCE node
I20260925 02:53:48.248714 18274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:48.249935 18274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:48.251223 18274 hybrid_clock.cc:648] HybridClock initialized: now 1790304828251151 us; error 63 us; skew 500 ppm
I20260925 02:53:48.251690 18274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:53:48 dist-test-slave-g1xh krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304828, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:48.255143 18274 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:53:48.257582 18274 webserver.cc:460] Webserver started at http://127.2.172.67:35813/ using document root <none> and password file <none>
I20260925 02:53:48.258585 18274 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:48.258755 18274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:48.264842 18274 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:53:48.268074 18289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:48.269650 18274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:53:48.269799 18274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e"
format_stamp: "Formatted at 2026-09-25 02:53:42 on dist-test-slave-g1xh"
server_key: "693c1955b114b37c5252973dcd95c1c4"
server_key_iv: "d7393604a2a6e9224860fcde41a03e1d"
server_key_version: "encryptionkey@0"
I20260925 02:53:48.270816 18274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:48.284439 18274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:48.288949 18274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:48.289194 18274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:48.290169 18274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:48.291533 18274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:48.291708 18274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:48.291792 18274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:48.291858 18274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:48.326679 18274 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:43437
I20260925 02:53:48.326766 18401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:43437 every 8 connection(s)
I20260925 02:53:48.328275 18274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:48.329568 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 18274
I20260925 02:53:48.333787 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:48.331793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:40079)
0925 02:53:48.332652 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:48.332890 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":329,"threads_started":1}
W20260925 02:53:48.334647 18402 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:42799 (0 consecutive failures): Network error: Failed to ping master at 127.2.172.126:42799: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
I20260925 02:53:48.335733 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:48.335156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:39469)
0925 02:53:48.335345 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:48.335417 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 02:53:48.336669 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:48.336307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:37313)
0925 02:53:48.336459 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:48.336555 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 02:53:48.336912 18402 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:53:48.396646 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:48.395827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:37461)
0925 02:53:48.396311 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:48.396419 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":238,"threads_started":1}
I20260925 02:53:48.789036 18407 raft_consensus.cc:482] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:48.789353 18407 raft_consensus.cc:504] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:48.790972 18407 leader_election.cc:288] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437), fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675)
Sep 25 02:53:48 dist-test-slave-g1xh krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304827, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
Sep 25 02:53:48 dist-test-slave-g1xh krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304827, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:53:48.822834 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:48.791697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:59423 (local address 127.2.172.67:43437)
0925 02:53:48.793435 (+ 1738us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:48.793451 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:48.793493 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:53:48.795769 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:48.795782 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:48.796307 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:48.796700 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:48.799483 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:48.801396 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:48.804276 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:48.805359 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:48.805573 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:48.810385 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:48.810398 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:48.810420 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:48.810470 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:48.818868 (+ 8398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:48.819849 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:48.819856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:48.819862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:48.820049 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:48.820491 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:48.820495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:48.820497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:48.821028 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:48.821227 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:48.822015 (+ 788us) server_negotiation.cc:299] Negotiation successful
0925 02:53:48.822363 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1551,"thread_start_us":1412,"threads_started":1}
I20260925 02:53:48.822875 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:48.791641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:54857 (local address 127.2.172.66:39675)
0925 02:53:48.793435 (+ 1794us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:48.793451 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:48.793493 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:53:48.795813 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:48.795823 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:48.796315 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:48.796700 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:48.799499 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:48.801404 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:48.804276 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:48.805359 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:48.805571 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:48.810816 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:48.810825 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:48.810836 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:48.810873 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:48.818595 (+ 7722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:48.819772 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:48.819780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:48.819786 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:48.819977 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:48.820465 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:48.820474 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:48.820476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:48.821023 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:48.821227 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:48.822017 (+ 790us) server_negotiation.cc:299] Negotiation successful
0925 02:53:48.822406 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1607,"thread_start_us":1488,"threads_started":1}
I20260925 02:53:48.822897 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:48.791406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39675 (local address 127.2.172.65:54857)
0925 02:53:48.792084 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:48.792145 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:48.792714 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:48.796893 (+ 4179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:48.796905 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:48.797924 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:48.799188 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:48.799218 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:48.801641 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:48.801647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:48.804084 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:48.804103 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:48.806222 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:48.806228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:48.806416 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:48.807080 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:48.807108 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:48.810537 (+ 3429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:48.818802 (+ 8265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:48.818812 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:48.818831 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:48.819530 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:48.820114 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:48.820120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:48.820123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:48.820330 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:48.821191 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:48.821196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:48.821517 (+ 321us) client_negotiation.cc:769] Sending connection context
0925 02:53:48.821796 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 02:53:48.822387 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":372,"spinlock_wait_cycles":31744,"thread_start_us":348,"threads_started":1}
I20260925 02:53:48.822971 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:48.791383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:43437 (local address 127.2.172.65:59423)
0925 02:53:48.792081 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:48.792139 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:48.792714 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:48.796893 (+ 4179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:48.796905 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:48.797924 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:48.799188 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:48.799218 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:48.801648 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:48.801651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:48.804084 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:48.804103 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:48.805585 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:48.805593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:48.806765 (+ 1172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:48.807364 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:48.807389 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:48.810032 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:48.819064 (+ 9032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:48.819069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:48.819072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:48.819678 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:48.820195 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:48.820204 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:48.820207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:48.820356 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:48.821182 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:48.821190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:48.821520 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 02:53:48.821793 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 02:53:48.822566 (+ 773us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":655,"spinlock_wait_cycles":62208}
W20260925 02:53:48.826670 18025 leader_election.cc:341] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675): Not found: Tablet not found: 78068476956c40d7b8dcaaeda8445ce7
W20260925 02:53:48.827056 18027 leader_election.cc:341] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437): Not found: Tablet not found: 78068476956c40d7b8dcaaeda8445ce7
I20260925 02:53:48.827230 18027 leader_election.cc:302] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94db12ef415843c0ad414ca2fdd6df2d; no voters: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e, fc7729d3bb6f4bf6a16ee6d54cad8604
I20260925 02:53:48.827816 18407 raft_consensus.cc:2726] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:53:48.874686 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:48.873601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:34911)
0925 02:53:48.874194 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:48.874336 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":336,"threads_started":1}
I20260925 02:53:49.338337 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:49.337541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:56175)
0925 02:53:49.338018 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:49.338128 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260925 02:53:49.398332 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:49.397621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:59801)
0925 02:53:49.398074 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:49.398161 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":221,"threads_started":1}
I20260925 02:53:49.876420 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:49.875738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:44113)
0925 02:53:49.876174 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:49.876260 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":221,"threads_started":1}
I20260925 02:53:50.340394 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:50.339449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:42891)
0925 02:53:50.340043 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:50.340182 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":358,"threads_started":1}
I20260925 02:53:50.400442 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:50.399339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:57121)
0925 02:53:50.400093 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:50.400213 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":394,"threads_started":1}
I20260925 02:53:50.878080 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:50.877316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:36751)
0925 02:53:50.877819 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:50.877905 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":265,"threads_started":1}
I20260925 02:53:50.918643 18418 raft_consensus.cc:482] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:50.918992 18418 raft_consensus.cc:504] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:50.919823 18418 leader_election.cc:288] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437), fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675)
W20260925 02:53:50.920910 18027 leader_election.cc:341] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437): Not found: Tablet not found: 78068476956c40d7b8dcaaeda8445ce7
W20260925 02:53:50.921120 18025 leader_election.cc:341] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675): Not found: Tablet not found: 78068476956c40d7b8dcaaeda8445ce7
I20260925 02:53:50.921296 18025 leader_election.cc:302] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94db12ef415843c0ad414ca2fdd6df2d; no voters: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e, fc7729d3bb6f4bf6a16ee6d54cad8604
I20260925 02:53:50.921578 18418 raft_consensus.cc:2726] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:53:51.342063 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:51.341310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:44347)
0925 02:53:51.341792 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:51.341885 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":263,"threads_started":1}
I20260925 02:53:51.402452 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:51.401518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:38785)
0925 02:53:51.402148 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:51.402255 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":326,"threads_started":1}
I20260925 02:53:51.880239 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:51.879166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:53433)
0925 02:53:51.879908 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:51.880025 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":435,"threads_started":1}
I20260925 02:53:52.344219 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:52.343119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:55629)
0925 02:53:52.343801 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:52.343917 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":255,"threads_started":1}
I20260925 02:53:52.404388 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:52.403618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:56855)
0925 02:53:52.404118 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:52.404204 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":242,"threads_started":1}
I20260925 02:53:52.698968 18424 raft_consensus.cc:482] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:52.699229 18424 raft_consensus.cc:504] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:52.699983 18424 leader_election.cc:288] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437), fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675)
W20260925 02:53:52.700848 18025 leader_election.cc:341] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675): Not found: Tablet not found: 78068476956c40d7b8dcaaeda8445ce7
W20260925 02:53:52.701025 18027 leader_election.cc:341] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437): Not found: Tablet not found: 78068476956c40d7b8dcaaeda8445ce7
I20260925 02:53:52.701159 18027 leader_election.cc:302] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94db12ef415843c0ad414ca2fdd6df2d; no voters: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e, fc7729d3bb6f4bf6a16ee6d54cad8604
I20260925 02:53:52.701402 18424 raft_consensus.cc:2726] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:53:52.881992 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:52.881244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:38045)
0925 02:53:52.881705 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:52.881796 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":234,"threads_started":1}
I20260925 02:53:53.346110 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:53.345374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:42317)
0925 02:53:53.345815 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:53.345913 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":230,"threads_started":1}
I20260925 02:53:53.406286 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:53.405416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:59643)
0925 02:53:53.405905 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:53.406049 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
I20260925 02:53:53.883805 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:53.882939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:49351)
0925 02:53:53.883427 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:53.883519 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":250,"threads_started":1}
I20260925 02:53:54.347978 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:54.347150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:38821)
0925 02:53:54.347680 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:54.347767 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":259,"threads_started":1}
I20260925 02:53:54.408277 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:54.407314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:50285)
0925 02:53:54.407914 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:54.408016 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":366,"threads_started":1}
I20260925 02:53:54.885696 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:54.884878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:40921)
0925 02:53:54.885412 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:54.885507 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":245,"threads_started":1}
I20260925 02:53:55.349882 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:55.349005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:42501)
0925 02:53:55.349582 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:55.349671 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":296,"threads_started":1}
I20260925 02:53:55.410171 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:55.409377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:39643)
0925 02:53:55.409875 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:55.410003 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":238,"threads_started":1}
I20260925 02:53:55.801285 18434 raft_consensus.cc:482] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:55.801573 18434 raft_consensus.cc:504] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:53:55.802280 18434 leader_election.cc:288] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437), fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675)
W20260925 02:53:55.803231 18027 leader_election.cc:341] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437): Not found: Tablet not found: 78068476956c40d7b8dcaaeda8445ce7
W20260925 02:53:55.803397 18025 leader_election.cc:341] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675): Not found: Tablet not found: 78068476956c40d7b8dcaaeda8445ce7
I20260925 02:53:55.803588 18025 leader_election.cc:302] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94db12ef415843c0ad414ca2fdd6df2d; no voters: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e, fc7729d3bb6f4bf6a16ee6d54cad8604
I20260925 02:53:55.803851 18434 raft_consensus.cc:2726] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:53:55.887661 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:55.886785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:55297)
0925 02:53:55.887318 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:55.887400 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":307,"threads_started":1}
I20260925 02:53:56.351640 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:56.350866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:49653)
0925 02:53:56.351285 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:56.351374 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":220,"threads_started":1}
I20260925 02:53:56.412106 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:56.411200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:54029)
0925 02:53:56.411759 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:56.411883 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":296,"threads_started":1}
I20260925 02:53:56.889366 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:56.888589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:34373)
0925 02:53:56.889120 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:56.889208 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":235,"threads_started":1}
I20260925 02:53:57.354173 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:57.352963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:38423)
0925 02:53:57.353728 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:57.353934 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":342,"threads_started":1}
I20260925 02:53:57.414414 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:57.413535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:48463)
0925 02:53:57.414088 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:57.414188 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
I20260925 02:53:57.891710 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:57.890672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:55161)
0925 02:53:57.891302 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:57.891404 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":349,"threads_started":1}
I20260925 02:53:58.356528 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:58.355648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:35005)
0925 02:53:58.356221 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:58.356307 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":337,"threads_started":1}
I20260925 02:53:58.416656 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:58.415765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:45469)
0925 02:53:58.416327 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:58.416446 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":277,"threads_started":1}
I20260925 02:53:58.893617 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:58.892779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:55945)
0925 02:53:58.893328 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:58.893436 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":280,"threads_started":1}
I20260925 02:53:59.358269 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:59.357518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:33139)
0925 02:53:59.358021 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:59.358106 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":248,"threads_started":1}
I20260925 02:53:59.418732 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:59.417815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:47589)
0925 02:53:59.418374 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:59.418518 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":326,"threads_started":1}
I20260925 02:53:59.895742 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:59.894802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:33209)
0925 02:53:59.895385 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:59.895498 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":320,"threads_started":1}
I20260925 02:54:00.360086 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:00.359285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:54537)
0925 02:54:00.359815 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:00.359903 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":299,"threads_started":1}
I20260925 02:54:00.420909 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:00.419950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:32793)
0925 02:54:00.420598 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:00.420723 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":327,"threads_started":1}
I20260925 02:54:00.685606 18450 raft_consensus.cc:482] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:54:00.685855 18450 raft_consensus.cc:504] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94db12ef415843c0ad414ca2fdd6df2d" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 38737 } } peers { permanent_uuid: "1bb7ccf8300d4b6d84ac3a7c8aa8f06e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 43437 } } peers { permanent_uuid: "fc7729d3bb6f4bf6a16ee6d54cad8604" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39675 } }
I20260925 02:54:00.686620 18450 leader_election.cc:288] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437), fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675)
W20260925 02:54:00.687769 18027 leader_election.cc:341] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bb7ccf8300d4b6d84ac3a7c8aa8f06e (127.2.172.67:43437): Not found: Tablet not found: 78068476956c40d7b8dcaaeda8445ce7
W20260925 02:54:00.687985 18025 leader_election.cc:341] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc7729d3bb6f4bf6a16ee6d54cad8604 (127.2.172.66:39675): Not found: Tablet not found: 78068476956c40d7b8dcaaeda8445ce7
I20260925 02:54:00.688112 18025 leader_election.cc:302] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94db12ef415843c0ad414ca2fdd6df2d; no voters: 1bb7ccf8300d4b6d84ac3a7c8aa8f06e, fc7729d3bb6f4bf6a16ee6d54cad8604
I20260925 02:54:00.688474 18450 raft_consensus.cc:2726] T 78068476956c40d7b8dcaaeda8445ce7 P 94db12ef415843c0ad414ca2fdd6df2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:54:00.897523 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:00.896669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:43199)
0925 02:54:00.897221 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:00.897324 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":283,"threads_started":1}
I20260925 02:54:01.361886 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:01.361141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:58489)
0925 02:54:01.361600 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:01.361698 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":234,"threads_started":1}
I20260925 02:54:01.423131 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:01.422197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:53931)
0925 02:54:01.422843 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:01.422951 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":347,"threads_started":1}
I20260925 02:54:01.899468 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:01.898582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:35629)
0925 02:54:01.899099 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:01.899203 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":293,"threads_started":1}
I20260925 02:54:02.363837 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:02.362927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:45447)
0925 02:54:02.363433 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:02.363541 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":251,"threads_started":1}
I20260925 02:54:02.425216 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:02.424495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:56343)
0925 02:54:02.424957 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:02.425049 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":224,"threads_started":1}
I20260925 02:54:02.901331 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:02.900546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.66:57249)
0925 02:54:02.901053 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:02.901174 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":289,"threads_started":1}
I20260925 02:54:03.339030 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 17974
I20260925 02:54:03.355630 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:03.340279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55294 (local address 127.2.172.65:38737)
0925 02:54:03.340842 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:03.340862 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:03.340894 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:54:03.340968 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:03.340977 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:03.341304 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:03.341483 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:03.343984 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:03.344910 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:03.346161 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:03.346721 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:03.346821 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:03.349496 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:03.349509 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:03.349523 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:03.349559 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:03.352583 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:03.353438 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:03.353446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:03.353448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:03.353621 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:03.354038 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:03.354042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:03.354044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:03.354433 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:03.354671 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:03.355070 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 02:54:03.355304 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":903,"threads_started":1}
I20260925 02:54:03.356963 18133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:54:03.366108 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:03.365205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.67:43095)
0925 02:54:03.365797 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:03.365945 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":330,"threads_started":1}
I20260925 02:54:03.449856 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:03.448779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42799 (local address 127.2.172.65:44859)
0925 02:54:03.449284 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:03.449517 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"threads_started":1}
I20260925 02:54:03.449820 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 17974
I20260925 02:54:03.470098 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 18143
I20260925 02:54:03.485301 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:03.470929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.2.172.66:39675)
0925 02:54:03.471467 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:03.471471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:03.471489 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:54:03.471546 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:03.471597 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:03.471675 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:03.471778 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:03.473177 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:03.473975 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:03.475536 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:03.476197 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:03.476329 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:03.479162 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:03.479168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:03.479171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:03.479224 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:03.482156 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:03.482958 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:03.482963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:03.482965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:03.483118 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:03.483520 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:03.483523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:03.483525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:03.483892 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:03.484121 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:03.484525 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 02:54:03.484719 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":334,"threads_started":1}
I20260925 02:54:03.486196 18266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:54:03.575865 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 18143
I20260925 02:54:03.594698 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 18274
I20260925 02:54:03.609432 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:03.595529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.2.172.67:43437)
0925 02:54:03.596198 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:03.596203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:03.596221 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:54:03.596280 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:03.596283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:03.596354 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:03.596450 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:03.597909 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:03.598720 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:03.600078 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:03.600542 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:03.600653 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:03.603759 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:03.603767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:03.603771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:03.603809 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:03.606670 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:03.607453 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:03.607457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:03.607459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:03.607684 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:03.608153 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:03.608157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:03.608159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:03.608432 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:03.608656 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:03.608934 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 02:54:03.609123 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":364,"threads_started":1}
I20260925 02:54:03.610246 18397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:54:03.696555 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 18274
2026-09-25T02:54:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24867 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:54:03 dist-test-slave-g1xh krb5kdc[18468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:54:03 dist-test-slave-g1xh krb5kdc[18468](info): set up 2 sockets
Sep 25 02:54:03 dist-test-slave-g1xh krb5kdc[18468](info): commencing operation
krb5kdc: starting...
W20260925 02:54:04.818719 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:54:04 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:54:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:54:05.083441 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:33377
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:54:05.419922 18481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:05.420210 18481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:05.420387 18481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:05.427564 18481 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:54:05.427659 18481 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:54:05.427736 18481 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:54:05.427814 18481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:54:05.427910 18481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:05.427997 18481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:05.428081 18481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:54:05.428169 18481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:54:05.438447 18481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:33377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:54:05.439832 18481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:05.441242 18481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:05.453405 18486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:05.453459 18487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:05.453887 18489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:05.454067 18481 server_base.cc:1034] running on GCE node
I20260925 02:54:05.454824 18481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:05.456316 18481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:05.457587 18481 hybrid_clock.cc:648] HybridClock initialized: now 1790304845457518 us; error 47 us; skew 500 ppm
I20260925 02:54:05.457933 18481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:54:05 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304845, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:05.461225 18481 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:54:05.462767 18481 webserver.cc:460] Webserver started at http://127.2.172.126:39331/ using document root <none> and password file <none>
I20260925 02:54:05.463471 18481 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:05.463629 18481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:05.463954 18481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:54:05.466919 18481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "b490732a5d124f1f999c40953816c7c6"
format_stamp: "Formatted at 2026-09-25 02:54:05 on dist-test-slave-g1xh"
server_key: "c7e45aebaf8a4bef6686efe69ba7ef07"
server_key_iv: "e9a1ec1f914d560a7202a52731c6e2ed"
server_key_version: "encryptionkey@0"
I20260925 02:54:05.467773 18481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "b490732a5d124f1f999c40953816c7c6"
format_stamp: "Formatted at 2026-09-25 02:54:05 on dist-test-slave-g1xh"
server_key: "c7e45aebaf8a4bef6686efe69ba7ef07"
server_key_iv: "e9a1ec1f914d560a7202a52731c6e2ed"
server_key_version: "encryptionkey@0"
I20260925 02:54:05.473068 18481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:54:05.476780 18496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:05.478189 18481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:54:05.478381 18481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "b490732a5d124f1f999c40953816c7c6"
format_stamp: "Formatted at 2026-09-25 02:54:05 on dist-test-slave-g1xh"
server_key: "c7e45aebaf8a4bef6686efe69ba7ef07"
server_key_iv: "e9a1ec1f914d560a7202a52731c6e2ed"
server_key_version: "encryptionkey@0"
I20260925 02:54:05.478617 18481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:05.493749 18481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:05.498322 18481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:05.498617 18481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:05.518872 18481 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:33377
I20260925 02:54:05.518841 18547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:33377 every 8 connection(s)
I20260925 02:54:05.520303 18481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:54:05.524125 18548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:54:05.529065 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 18481
I20260925 02:54:05.529337 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:54:05.530206 2737 external_mini_cluster.cc:1428] Setting key edce70c185a061c54cacc5ccb18dc52d
I20260925 02:54:05.536564 18548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Bootstrap starting.
I20260925 02:54:05.539865 18548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Neither blocks nor log segments found. Creating new log.
I20260925 02:54:05.540905 18548 log.cc:824] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:05.544473 18548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: No bootstrap required, opened a new log
Sep 25 02:54:05 dist-test-slave-g1xh krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:54:05.555631 18548 raft_consensus.cc:348] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } }
I20260925 02:54:05.556167 18548 raft_consensus.cc:374] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:54:05.556284 18548 raft_consensus.cc:729] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b490732a5d124f1f999c40953816c7c6, State: Initialized, Role: FOLLOWER
I20260925 02:54:05.556998 18548 consensus_queue.cc:260] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } }
I20260925 02:54:05.557288 18548 raft_consensus.cc:388] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:54:05.557471 18548 raft_consensus.cc:482] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:54:05.557621 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:05.532445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.2.172.126:33377)
0925 02:54:05.533393 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:05.533405 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:05.533449 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:54:05.535321 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:05.535347 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:05.535824 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:05.536269 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:05.537880 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:05.539341 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:05.540783 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:05.541650 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:05.541858 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:05.546462 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:05.546472 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:05.546496 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:05.546547 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:05.550996 (+ 4449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:05.552530 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:05.552540 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:05.552549 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:05.552765 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:05.553279 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:05.553286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:05.553290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:05.554065 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:05.554433 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:05.555648 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0925 02:54:05.556215 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":367,"threads_started":1}
I20260925 02:54:05.557670 18548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:54:05.560956 18548 raft_consensus.cc:504] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } }
I20260925 02:54:05.561390 18548 leader_election.cc:302] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b490732a5d124f1f999c40953816c7c6; no voters:
I20260925 02:54:05.561905 18548 leader_election.cc:288] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:54:05.562074 18553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:54:05.562752 18553 raft_consensus.cc:686] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 1 LEADER]: Becoming Leader. State: Replica: b490732a5d124f1f999c40953816c7c6, State: Running, Role: LEADER
I20260925 02:54:05.563303 18553 consensus_queue.cc:237] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } }
I20260925 02:54:05.564102 18548 sys_catalog.cc:564] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:54:05.567085 18555 sys_catalog.cc:455] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b490732a5d124f1f999c40953816c7c6. Latest consensus state: current_term: 1 leader_uuid: "b490732a5d124f1f999c40953816c7c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } } }
I20260925 02:54:05.567541 18555 sys_catalog.cc:458] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:54:05.573843 18560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:54:05.573626 18554 sys_catalog.cc:455] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b490732a5d124f1f999c40953816c7c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } } }
I20260925 02:54:05.574059 18554 sys_catalog.cc:458] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:54:05.582314 18560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:54:05.591254 18560 catalog_manager.cc:1347] Generated new cluster ID: 8a0da812999743db95af613aac785841
I20260925 02:54:05.591463 18560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:54:05.608194 18560 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:54:05.610188 18560 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:54:05.620687 18560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Generated new TSK 0
I20260925 02:54:05.621774 18560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:54:05.697821 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:33377
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:54:06.037850 18574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:06.038132 18574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:06.038300 18574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:06.045814 18574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:06.045933 18574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:06.046051 18574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:54:06.056327 18574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:33377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:54:06.057585 18574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:06.058820 18574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:06.072345 18582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:06.072409 18579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:06.072438 18580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:06.073588 18574 server_base.cc:1034] running on GCE node
I20260925 02:54:06.074218 18574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:06.075270 18574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:06.076525 18574 hybrid_clock.cc:648] HybridClock initialized: now 1790304846076487 us; error 46 us; skew 500 ppm
I20260925 02:54:06.076896 18574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:54:06 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:06.080266 18574 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:54:06.081701 18574 webserver.cc:460] Webserver started at http://127.2.172.65:36425/ using document root <none> and password file <none>
I20260925 02:54:06.082576 18574 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:06.082684 18574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:06.083006 18574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:54:06.086062 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "a7694f1cbccf47178140c221f9414f22"
format_stamp: "Formatted at 2026-09-25 02:54:06 on dist-test-slave-g1xh"
server_key: "06b474b1498d68f7de6c2a207a17935f"
server_key_iv: "737fa3fa2e545574e2666ac675f4153e"
server_key_version: "encryptionkey@0"
I20260925 02:54:06.086777 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "a7694f1cbccf47178140c221f9414f22"
format_stamp: "Formatted at 2026-09-25 02:54:06 on dist-test-slave-g1xh"
server_key: "06b474b1498d68f7de6c2a207a17935f"
server_key_iv: "737fa3fa2e545574e2666ac675f4153e"
server_key_version: "encryptionkey@0"
I20260925 02:54:06.092381 18574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 02:54:06.096218 18589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:06.097800 18574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:54:06.097999 18574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "a7694f1cbccf47178140c221f9414f22"
format_stamp: "Formatted at 2026-09-25 02:54:06 on dist-test-slave-g1xh"
server_key: "06b474b1498d68f7de6c2a207a17935f"
server_key_iv: "737fa3fa2e545574e2666ac675f4153e"
server_key_version: "encryptionkey@0"
I20260925 02:54:06.098202 18574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:06.113243 18574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:06.118119 18574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:06.118461 18574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:06.119644 18574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:54:06.121225 18574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:54:06.121312 18574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:06.121452 18574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:54:06.121536 18574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:06.157541 18574 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:36243
I20260925 02:54:06.157586 18701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:36243 every 8 connection(s)
I20260925 02:54:06.158955 18574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:54:06.163956 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 18574
I20260925 02:54:06.164194 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:54:06.165113 2737 external_mini_cluster.cc:1428] Setting key 2c9e5e9b63a742ddf446000a503db975
Sep 25 02:54:06 dist-test-slave-g1xh krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:54:06.185956 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:06.161840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:47861 (local address 127.2.172.126:33377)
0925 02:54:06.162322 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:06.162328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:06.163383 (+ 1055us) server_negotiation.cc:407] Connection header received
0925 02:54:06.166379 (+ 2996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:06.166386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:06.166476 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:06.166601 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:06.168767 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:06.169484 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:06.172445 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:06.172877 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:06.172980 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:06.178043 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:06.178054 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:06.178061 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:06.178117 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:06.181919 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:06.183023 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:06.183032 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:06.183038 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:06.183204 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:06.183821 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:06.183829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:06.183836 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:06.184239 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:06.184406 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:06.185180 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 02:54:06.185471 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":222,"threads_started":1}
I20260925 02:54:06.186712 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:06.162457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:47861)
0925 02:54:06.163203 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:06.163249 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:06.165986 (+ 2737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:06.166820 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:06.166830 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:06.167681 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:54:06.168584 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:06.168615 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:06.169650 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:06.169659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:06.172249 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:06.172270 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:06.173137 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:06.173147 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:06.173523 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:06.174355 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:54:06.174394 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:54:06.177713 (+ 3319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:54:06.182127 (+ 4414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:06.182135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:06.182151 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:06.182875 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:06.183344 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:06.183347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:06.183350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:06.183686 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:06.184372 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:54:06.184379 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:54:06.184740 (+ 361us) client_negotiation.cc:769] Sending connection context
0925 02:54:06.185002 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 02:54:06.185451 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":299,"threads_started":1}
I20260925 02:54:06.188519 18702 heartbeater.cc:347] Connected to a master server at 127.2.172.126:33377
I20260925 02:54:06.188853 18702 heartbeater.cc:464] Registering TS with master...
I20260925 02:54:06.189633 18702 heartbeater.cc:511] Master 127.2.172.126:33377 requested a full tablet report, sending...
I20260925 02:54:06.192212 18513 ts_manager.cc:194] Registered new tserver with Master: a7694f1cbccf47178140c221f9414f22 (127.2.172.65:36243)
I20260925 02:54:06.194494 18513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:47861
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:54:06.231247 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:33377
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:54:06.574431 18708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:06.574735 18708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:06.574898 18708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:06.582454 18708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:06.582692 18708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:06.582891 18708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:54:06.593307 18708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:33377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:54:06.594637 18708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:06.596038 18708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:06.609947 18714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:06.609972 18716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:06.610459 18713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:06.610921 18708 server_base.cc:1034] running on GCE node
I20260925 02:54:06.611663 18708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:06.612696 18708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:06.613946 18708 hybrid_clock.cc:648] HybridClock initialized: now 1790304846613906 us; error 93 us; skew 500 ppm
I20260925 02:54:06.614277 18708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:54:06 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:06.617594 18708 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:54:06.619225 18708 webserver.cc:460] Webserver started at http://127.2.172.66:33799/ using document root <none> and password file <none>
I20260925 02:54:06.620033 18708 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:06.620180 18708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:06.620504 18708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:54:06.623412 18708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "57b40c64bfbf403cbaa1d2b4bc30d271"
format_stamp: "Formatted at 2026-09-25 02:54:06 on dist-test-slave-g1xh"
server_key: "1d1a859448e3c39abb2203da6a690119"
server_key_iv: "78906020cd91e52628baf7841cd847e5"
server_key_version: "encryptionkey@0"
I20260925 02:54:06.624272 18708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "57b40c64bfbf403cbaa1d2b4bc30d271"
format_stamp: "Formatted at 2026-09-25 02:54:06 on dist-test-slave-g1xh"
server_key: "1d1a859448e3c39abb2203da6a690119"
server_key_iv: "78906020cd91e52628baf7841cd847e5"
server_key_version: "encryptionkey@0"
I20260925 02:54:06.629668 18708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:54:06.633335 18723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:06.634682 18708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:54:06.634876 18708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "57b40c64bfbf403cbaa1d2b4bc30d271"
format_stamp: "Formatted at 2026-09-25 02:54:06 on dist-test-slave-g1xh"
server_key: "1d1a859448e3c39abb2203da6a690119"
server_key_iv: "78906020cd91e52628baf7841cd847e5"
server_key_version: "encryptionkey@0"
I20260925 02:54:06.635098 18708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:06.651916 18708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:06.656277 18708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:06.656598 18708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:06.657642 18708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:54:06.659006 18708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:54:06.659114 18708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:06.659308 18708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:54:06.659385 18708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:06.693948 18708 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:37741
I20260925 02:54:06.694003 18835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:37741 every 8 connection(s)
I20260925 02:54:06.695225 18708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:54:06.698582 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 18708
I20260925 02:54:06.698801 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:54:06.699262 2737 external_mini_cluster.cc:1428] Setting key 3730afbe62c9e9b0910829f040432b33
Sep 25 02:54:06 dist-test-slave-g1xh krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:54:06.715839 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:06.697854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:59477 (local address 127.2.172.126:33377)
0925 02:54:06.698351 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:06.698358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:06.699424 (+ 1066us) server_negotiation.cc:407] Connection header received
0925 02:54:06.701266 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:06.701272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:06.701359 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:06.701478 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:06.703026 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:06.703843 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:06.705380 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:06.705802 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:06.705938 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:06.709731 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:06.709737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:06.709740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:06.709774 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:06.712603 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:06.713414 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:06.713420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:06.713425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:06.713558 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:06.714034 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:06.714041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:06.714044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:06.714312 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:06.714468 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:06.715172 (+ 704us) server_negotiation.cc:299] Negotiation successful
0925 02:54:06.715402 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":222,"threads_started":1}
I20260925 02:54:06.716866 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:06.698394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:59477)
0925 02:54:06.699203 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:06.699261 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:06.700985 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:06.701746 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:06.701753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:06.702294 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:54:06.702848 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:06.702865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:06.704002 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:06.704008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:06.705215 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:06.705227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:06.706025 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:06.706034 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:06.706266 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:06.706944 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:54:06.706966 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:54:06.709431 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:54:06.712798 (+ 3367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:06.712807 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:06.712827 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:06.713267 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:06.713674 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:06.713679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:06.713683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:06.713913 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:06.714445 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:54:06.714453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:54:06.714814 (+ 361us) client_negotiation.cc:769] Sending connection context
0925 02:54:06.715091 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 02:54:06.715680 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":346,"threads_started":1}
I20260925 02:54:06.718518 18836 heartbeater.cc:347] Connected to a master server at 127.2.172.126:33377
I20260925 02:54:06.718855 18836 heartbeater.cc:464] Registering TS with master...
I20260925 02:54:06.719821 18836 heartbeater.cc:511] Master 127.2.172.126:33377 requested a full tablet report, sending...
I20260925 02:54:06.721647 18513 ts_manager.cc:194] Registered new tserver with Master: 57b40c64bfbf403cbaa1d2b4bc30d271 (127.2.172.66:37741)
I20260925 02:54:06.722904 18513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:59477
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:54:06.758440 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:33377
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:54:07.092751 18842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:07.093053 18842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:07.093151 18842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:07.100739 18842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:07.100875 18842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:07.100992 18842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:54:07.111100 18842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:33377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:54:07.112433 18842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:07.113724 18842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:07.127135 18847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:07.127122 18848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:07.127079 18850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:07.128158 18842 server_base.cc:1034] running on GCE node
I20260925 02:54:07.128753 18842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:07.129662 18842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:07.130893 18842 hybrid_clock.cc:648] HybridClock initialized: now 1790304847130843 us; error 59 us; skew 500 ppm
I20260925 02:54:07.131209 18842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:54:07 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304847, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:07.134534 18842 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:54:07.136072 18842 webserver.cc:460] Webserver started at http://127.2.172.67:43015/ using document root <none> and password file <none>
I20260925 02:54:07.136825 18842 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:07.136933 18842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:07.137198 18842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:54:07.140275 18842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "6be9db95a79f4e90baccabb7428c9451"
format_stamp: "Formatted at 2026-09-25 02:54:07 on dist-test-slave-g1xh"
server_key: "f9088126d5d5d908841e51e5fa1f03e0"
server_key_iv: "a331b8599f456943abf25fef8502a7ff"
server_key_version: "encryptionkey@0"
I20260925 02:54:07.141001 18842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "6be9db95a79f4e90baccabb7428c9451"
format_stamp: "Formatted at 2026-09-25 02:54:07 on dist-test-slave-g1xh"
server_key: "f9088126d5d5d908841e51e5fa1f03e0"
server_key_iv: "a331b8599f456943abf25fef8502a7ff"
server_key_version: "encryptionkey@0"
I20260925 02:54:07.146371 18842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:54:07.150197 18857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:07.151638 18842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:54:07.151783 18842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "6be9db95a79f4e90baccabb7428c9451"
format_stamp: "Formatted at 2026-09-25 02:54:07 on dist-test-slave-g1xh"
server_key: "f9088126d5d5d908841e51e5fa1f03e0"
server_key_iv: "a331b8599f456943abf25fef8502a7ff"
server_key_version: "encryptionkey@0"
I20260925 02:54:07.151964 18842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:07.190798 18842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:07.195253 18842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:07.195649 18842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:07.196776 18842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:54:07.197800 18702 heartbeater.cc:503] Master 127.2.172.126:33377 was elected leader, sending a full tablet report...
I20260925 02:54:07.198459 18842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:54:07.198575 18842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:07.198728 18842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:54:07.198807 18842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:07.234841 18842 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:38617
I20260925 02:54:07.234898 18969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:38617 every 8 connection(s)
I20260925 02:54:07.236292 18842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:54:07.243829 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 18842
I20260925 02:54:07.244042 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:54:07.244565 2737 external_mini_cluster.cc:1428] Setting key d322ab0cfffff322ae347bcfd03529ca
Sep 25 02:54:07 dist-test-slave-g1xh krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304847, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:54:07.258028 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.239019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:34963 (local address 127.2.172.126:33377)
0925 02:54:07.239595 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:07.239602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:07.240643 (+ 1041us) server_negotiation.cc:407] Connection header received
0925 02:54:07.242528 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:07.242532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:07.242607 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:07.242702 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:07.244501 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.245558 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.247527 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.248136 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.248285 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.252079 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:07.252083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:07.252087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:07.252119 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:07.255040 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:07.255921 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:07.255925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:07.255927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:07.256041 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:07.256514 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:07.256517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:07.256519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:07.256776 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:07.256927 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:07.257589 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 02:54:07.257750 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":286,"threads_started":1}
I20260925 02:54:07.259331 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.239701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:34963)
0925 02:54:07.240486 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:07.240530 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:07.242273 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:07.242922 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:07.242930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:07.243530 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:54:07.244297 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:07.244326 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.245728 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.245734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:07.247361 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:07.247374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.248411 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.248420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:07.248666 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.249346 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:54:07.249377 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:54:07.251861 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:54:07.255181 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:07.255189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:07.255203 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:07.255766 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:07.256162 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:07.256170 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:07.256173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:07.256397 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:07.256898 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:54:07.256903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:54:07.257229 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 02:54:07.257486 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 02:54:07.257894 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":282,"threads_started":1}
I20260925 02:54:07.260828 18970 heartbeater.cc:347] Connected to a master server at 127.2.172.126:33377
I20260925 02:54:07.261135 18970 heartbeater.cc:464] Registering TS with master...
I20260925 02:54:07.261832 18970 heartbeater.cc:511] Master 127.2.172.126:33377 requested a full tablet report, sending...
I20260925 02:54:07.263245 18513 ts_manager.cc:194] Registered new tserver with Master: 6be9db95a79f4e90baccabb7428c9451 (127.2.172.67:38617)
I20260925 02:54:07.264123 18513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:34963
I20260925 02:54:07.272657 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:54:07.289873 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.277075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49980 (local address 127.2.172.126:33377)
0925 02:54:07.277242 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:07.277246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:07.277738 (+ 492us) server_negotiation.cc:407] Connection header received
0925 02:54:07.277862 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:07.277865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:07.277929 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:07.278013 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:07.279538 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.280355 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.281862 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.282261 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.282400 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.284778 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:07.284784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:07.284788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:07.284822 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:07.287487 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:07.288190 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:07.288194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:07.288197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:07.288353 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:07.288669 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:07.288672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:07.288674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:07.288992 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:07.289167 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:07.289450 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 02:54:07.289619 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:54:07.293819 18513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:54:07.300884 18513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:54:07.352246 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.314918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:36243 (local address 127.0.0.1:33794)
0925 02:54:07.342082 (+ 27164us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:07.342119 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:07.342335 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:07.344560 (+ 2225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:07.344567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:07.344606 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:07.345047 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:07.345057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.347076 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.347082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:07.349354 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:07.349366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.351082 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.351088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:07.351299 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.351355 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 02:54:07.351469 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:54:07.351742 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26973,"mutex_wait_us":29,"thread_start_us":27542,"threads_started":1}
I20260925 02:54:07.352723 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.315011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.2.172.65:36243)
0925 02:54:07.343890 (+ 28879us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:07.343897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:07.343920 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:54:07.344004 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:07.344011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:07.344286 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:07.345217 (+ 931us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:07.345417 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.346925 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.349522 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.350824 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.351938 (+ 1114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.352164 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:07.352240 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 02:54:07.352409 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28686,"thread_start_us":28466,"threads_started":1}
I20260925 02:54:07.352878 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.315476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:37741 (local address 127.0.0.1:53152)
0925 02:54:07.343601 (+ 28125us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:07.343626 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:07.344272 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:07.345220 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:07.345224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:07.345250 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:07.345702 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:07.345714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.347521 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.347527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:07.349920 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:07.349935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.352007 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.352013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:07.352205 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.352253 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 02:54:07.352355 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:54:07.352497 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27967,"mutex_wait_us":76,"thread_start_us":7991,"threads_started":1}
I20260925 02:54:07.353132 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.316093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.2.172.66:37741)
0925 02:54:07.341173 (+ 25080us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:07.341188 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:07.343731 (+ 2543us) server_negotiation.cc:407] Connection header received
0925 02:54:07.344548 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:07.344557 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:07.344767 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:07.344940 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:07.345881 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.347375 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.350087 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.351509 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.351662 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.351861 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:07.352621 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 02:54:07.352790 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24803,"thread_start_us":25129,"threads_started":1}
I20260925 02:54:07.353257 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.314897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:38617 (local address 127.0.0.1:53090)
0925 02:54:07.324194 (+ 9297us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:07.324243 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:07.324466 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:07.344627 (+ 20161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:07.344632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:07.344658 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:07.345208 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:07.345219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.347039 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.347044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:07.349920 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:07.349935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.351584 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.351592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:07.351801 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.351832 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:54:07.351915 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:54:07.352048 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9037,"mutex_wait_us":88,"thread_start_us":28353,"threads_started":1}
I20260925 02:54:07.354763 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.315589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.2.172.67:38617)
0925 02:54:07.344048 (+ 28459us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:07.344056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:07.344084 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:54:07.344181 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:07.344188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:07.344421 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:07.345423 (+ 1002us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:07.345705 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.346880 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.350131 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.351321 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.354042 (+ 2721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.354303 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:07.354374 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 02:54:07.354513 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28286,"thread_start_us":27464,"threads_started":1}
I20260925 02:54:07.356938 18771 tablet_service.cc:1467] Processing CreateTablet for tablet 1673e73b396c494fbb4b2d0e976a5ec8 (DEFAULT_TABLE table=test-table [id=26e2f5b133b549a4a6705aa47ca9644a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:54:07.356938 18637 tablet_service.cc:1467] Processing CreateTablet for tablet 1673e73b396c494fbb4b2d0e976a5ec8 (DEFAULT_TABLE table=test-table [id=26e2f5b133b549a4a6705aa47ca9644a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:54:07.358440 18637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1673e73b396c494fbb4b2d0e976a5ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:54:07.358438 18771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1673e73b396c494fbb4b2d0e976a5ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:54:07.358215 18905 tablet_service.cc:1467] Processing CreateTablet for tablet 1673e73b396c494fbb4b2d0e976a5ec8 (DEFAULT_TABLE table=test-table [id=26e2f5b133b549a4a6705aa47ca9644a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:54:07.359678 18905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1673e73b396c494fbb4b2d0e976a5ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:54:07.375602 18990 tablet_bootstrap.cc:492] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271: Bootstrap starting.
I20260925 02:54:07.376194 18991 tablet_bootstrap.cc:492] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22: Bootstrap starting.
I20260925 02:54:07.377674 18989 tablet_bootstrap.cc:492] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451: Bootstrap starting.
I20260925 02:54:07.379799 18990 tablet_bootstrap.cc:654] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271: Neither blocks nor log segments found. Creating new log.
I20260925 02:54:07.379901 18991 tablet_bootstrap.cc:654] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22: Neither blocks nor log segments found. Creating new log.
I20260925 02:54:07.380873 18990 log.cc:824] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:07.381139 18991 log.cc:824] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:07.381179 18989 tablet_bootstrap.cc:654] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451: Neither blocks nor log segments found. Creating new log.
I20260925 02:54:07.382431 18989 log.cc:824] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:07.384473 18991 tablet_bootstrap.cc:492] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22: No bootstrap required, opened a new log
I20260925 02:54:07.384645 18990 tablet_bootstrap.cc:492] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271: No bootstrap required, opened a new log
I20260925 02:54:07.384778 18991 ts_tablet_manager.cc:1397] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 02:54:07.385002 18990 ts_tablet_manager.cc:1397] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260925 02:54:07.390443 18989 tablet_bootstrap.cc:492] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451: No bootstrap required, opened a new log
I20260925 02:54:07.390749 18989 ts_tablet_manager.cc:1397] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260925 02:54:07.392511 18991 raft_consensus.cc:348] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } } peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } }
I20260925 02:54:07.392890 18991 raft_consensus.cc:374] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:54:07.393054 18991 raft_consensus.cc:729] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7694f1cbccf47178140c221f9414f22, State: Initialized, Role: FOLLOWER
I20260925 02:54:07.393815 18991 consensus_queue.cc:260] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } } peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } }
I20260925 02:54:07.395964 18990 raft_consensus.cc:348] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } } peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } }
I20260925 02:54:07.396425 18991 ts_tablet_manager.cc:1428] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260925 02:54:07.396466 18990 raft_consensus.cc:374] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:54:07.396631 18990 raft_consensus.cc:729] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b40c64bfbf403cbaa1d2b4bc30d271, State: Initialized, Role: FOLLOWER
I20260925 02:54:07.397292 18990 consensus_queue.cc:260] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } } peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } }
I20260925 02:54:07.398415 18836 heartbeater.cc:503] Master 127.2.172.126:33377 was elected leader, sending a full tablet report...
I20260925 02:54:07.399091 18990 ts_tablet_manager.cc:1428] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 02:54:07.401340 18989 raft_consensus.cc:348] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } } peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } }
I20260925 02:54:07.401831 18989 raft_consensus.cc:374] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:54:07.401950 18989 raft_consensus.cc:729] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be9db95a79f4e90baccabb7428c9451, State: Initialized, Role: FOLLOWER
I20260925 02:54:07.402681 18989 consensus_queue.cc:260] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } } peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } }
I20260925 02:54:07.403895 18970 heartbeater.cc:503] Master 127.2.172.126:33377 was elected leader, sending a full tablet report...
I20260925 02:54:07.404582 18989 ts_tablet_manager.cc:1428] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
W20260925 02:54:07.411676 18703 tablet.cc:2369] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:54:07.447317 18837 tablet.cc:2369] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:54:07.484175 18997 raft_consensus.cc:482] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:54:07.484503 18997 raft_consensus.cc:504] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } } peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } }
I20260925 02:54:07.486196 18997 leader_election.cc:288] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7694f1cbccf47178140c221f9414f22 (127.2.172.65:36243), 57b40c64bfbf403cbaa1d2b4bc30d271 (127.2.172.66:37741)
W20260925 02:54:07.488125 18971 tablet.cc:2369] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:54:07.492702 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.486621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:36243 (local address 127.2.172.67:51161)
0925 02:54:07.486876 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:07.486899 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:07.487087 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:07.487566 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:07.487572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:07.487602 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:07.487789 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:07.487797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.489010 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.489015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:07.490869 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:07.490884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.492166 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.492170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:07.492305 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.492329 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:54:07.492409 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:54:07.492501 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":25}
I20260925 02:54:07.492858 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.486637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:51161 (local address 127.2.172.65:36243)
0925 02:54:07.486907 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:07.486915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:07.487009 (+ 94us) server_negotiation.cc:407] Connection header received
0925 02:54:07.487307 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:07.487313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:07.487416 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:07.487519 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:07.487956 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.488870 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.491024 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.492023 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.492180 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.492256 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:07.492461 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 02:54:07.492607 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260925 02:54:07.493736 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.487240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:42057 (local address 127.2.172.66:37741)
0925 02:54:07.487457 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:07.487463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:07.487672 (+ 209us) server_negotiation.cc:407] Connection header received
0925 02:54:07.487908 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:07.487913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:07.487990 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:07.488087 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:07.488816 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.489545 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.491624 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.492811 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.492974 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.493046 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:07.493417 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 02:54:07.493549 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:54:07.493894 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.486653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:37741 (local address 127.2.172.67:42057)
0925 02:54:07.487523 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:07.487536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:07.487746 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:07.488126 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:07.488131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:07.488159 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:07.488615 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:07.488627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.489691 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.489696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:07.491467 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:07.491478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.493141 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.493145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:07.493262 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.493285 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:54:07.493365 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:54:07.493511 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":25,"thread_start_us":489,"threads_started":1}
I20260925 02:54:07.493960 18657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1673e73b396c494fbb4b2d0e976a5ec8" candidate_uuid: "6be9db95a79f4e90baccabb7428c9451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7694f1cbccf47178140c221f9414f22" is_pre_election: true
I20260925 02:54:07.494624 18657 raft_consensus.cc:2454] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6be9db95a79f4e90baccabb7428c9451 in term 0.
I20260925 02:54:07.495054 18791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1673e73b396c494fbb4b2d0e976a5ec8" candidate_uuid: "6be9db95a79f4e90baccabb7428c9451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" is_pre_election: true
I20260925 02:54:07.495280 18859 leader_election.cc:302] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6be9db95a79f4e90baccabb7428c9451, a7694f1cbccf47178140c221f9414f22; no voters:
I20260925 02:54:07.495622 18791 raft_consensus.cc:2454] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6be9db95a79f4e90baccabb7428c9451 in term 0.
I20260925 02:54:07.495697 18997 raft_consensus.cc:2781] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:54:07.495834 18997 raft_consensus.cc:482] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:54:07.495950 18997 raft_consensus.cc:3037] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:54:07.498760 18997 raft_consensus.cc:504] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } } peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } }
I20260925 02:54:07.499439 18997 leader_election.cc:288] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [CANDIDATE]: Term 1 election: Requested vote from peers a7694f1cbccf47178140c221f9414f22 (127.2.172.65:36243), 57b40c64bfbf403cbaa1d2b4bc30d271 (127.2.172.66:37741)
I20260925 02:54:07.499858 18657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1673e73b396c494fbb4b2d0e976a5ec8" candidate_uuid: "6be9db95a79f4e90baccabb7428c9451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7694f1cbccf47178140c221f9414f22"
I20260925 02:54:07.499882 18791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1673e73b396c494fbb4b2d0e976a5ec8" candidate_uuid: "6be9db95a79f4e90baccabb7428c9451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271"
I20260925 02:54:07.500074 18657 raft_consensus.cc:3037] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:54:07.500072 18791 raft_consensus.cc:3037] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:54:07.502436 18791 raft_consensus.cc:2454] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be9db95a79f4e90baccabb7428c9451 in term 1.
I20260925 02:54:07.502569 18657 raft_consensus.cc:2454] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be9db95a79f4e90baccabb7428c9451 in term 1.
I20260925 02:54:07.502980 18858 leader_election.cc:302] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57b40c64bfbf403cbaa1d2b4bc30d271, 6be9db95a79f4e90baccabb7428c9451; no voters:
I20260925 02:54:07.503361 18997 raft_consensus.cc:2781] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:54:07.503810 18997 raft_consensus.cc:686] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 1 LEADER]: Becoming Leader. State: Replica: 6be9db95a79f4e90baccabb7428c9451, State: Running, Role: LEADER
I20260925 02:54:07.504364 18997 consensus_queue.cc:237] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } } peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } }
I20260925 02:54:07.508960 18512 catalog_manager.cc:5495] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6be9db95a79f4e90baccabb7428c9451 (127.2.172.67). New cstate: current_term: 1 leader_uuid: "6be9db95a79f4e90baccabb7428c9451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } health_report { overall_health: UNKNOWN } } }
I20260925 02:54:07.548801 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.543106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.2.172.67:38617)
0925 02:54:07.543358 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:07.543364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:07.543383 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:54:07.543614 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:07.543628 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:07.543718 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:07.543887 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:54:07.544275 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.544995 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.546500 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.546901 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.547039 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.547604 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:54:07.548187 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:54:07.548358 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:07.548410 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 02:54:07.548494 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:54:07.560352 18657 raft_consensus.cc:1261] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22 [term 1 FOLLOWER]: Refusing update from remote peer 6be9db95a79f4e90baccabb7428c9451: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:54:07.560360 18791 raft_consensus.cc:1261] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 1 FOLLOWER]: Refusing update from remote peer 6be9db95a79f4e90baccabb7428c9451: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:54:07.561291 18997 consensus_queue.cc:1035] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:54:07.561590 18999 consensus_queue.cc:1035] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:54:07.574095 19003 mvcc.cc:205] Tried to move back new op lower bound from 7333088655597232128 to 7333088655382040576. Current Snapshot: MvccSnapshot[applied={T|T < 7333088655597232128}]
I20260925 02:54:07.589895 18513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:54:07.590194 18513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:54:07.593741 18513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Sending DeleteTablet for 3 replicas of tablet 1673e73b396c494fbb4b2d0e976a5ec8
I20260925 02:54:07.594978 18637 tablet_service.cc:1514] Processing DeleteTablet for tablet 1673e73b396c494fbb4b2d0e976a5ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:54:07 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:33794
I20260925 02:54:07.594981 18771 tablet_service.cc:1514] Processing DeleteTablet for tablet 1673e73b396c494fbb4b2d0e976a5ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:54:07 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:53152
I20260925 02:54:07.595009 18905 tablet_service.cc:1514] Processing DeleteTablet for tablet 1673e73b396c494fbb4b2d0e976a5ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:54:07 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:53090
I20260925 02:54:07.597142 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 18574
I20260925 02:54:07.607679 19012 tablet_replica.cc:330] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22: stopping tablet replica
I20260925 02:54:07.607883 19014 tablet_replica.cc:330] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451: stopping tablet replica
I20260925 02:54:07.608321 19013 tablet_replica.cc:330] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271: stopping tablet replica
I20260925 02:54:07.608523 19014 raft_consensus.cc:2229] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:54:07.608901 19013 raft_consensus.cc:2229] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:54:07.609287 19014 raft_consensus.cc:2258] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:54:07.609419 19012 raft_consensus.cc:2229] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:54:07.610019 19013 raft_consensus.cc:2258] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:54:07.610512 19012 raft_consensus.cc:2258] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:54:07.612221 19014 ts_tablet_manager.cc:1905] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:54:07.612901 19012 ts_tablet_manager.cc:1905] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:54:07.613523 19013 ts_tablet_manager.cc:1905] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:54:07.620641 19012 ts_tablet_manager.cc:1918] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:54:07.620874 19012 log.cc:1196] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/1673e73b396c494fbb4b2d0e976a5ec8
I20260925 02:54:07.621445 19012 ts_tablet_manager.cc:1939] T 1673e73b396c494fbb4b2d0e976a5ec8 P a7694f1cbccf47178140c221f9414f22: Deleting consensus metadata
I20260925 02:54:07.623103 19014 ts_tablet_manager.cc:1918] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:54:07.623366 19014 log.cc:1196] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/1673e73b396c494fbb4b2d0e976a5ec8
I20260925 02:54:07.623387 18498 catalog_manager.cc:4841] TS a7694f1cbccf47178140c221f9414f22 (127.2.172.65:36243): tablet 1673e73b396c494fbb4b2d0e976a5ec8 (table test-table [id=26e2f5b133b549a4a6705aa47ca9644a]) successfully deleted
I20260925 02:54:07.624065 19014 ts_tablet_manager.cc:1939] T 1673e73b396c494fbb4b2d0e976a5ec8 P 6be9db95a79f4e90baccabb7428c9451: Deleting consensus metadata
I20260925 02:54:07.624233 19013 ts_tablet_manager.cc:1918] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:54:07.624477 19013 log.cc:1196] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/1673e73b396c494fbb4b2d0e976a5ec8
I20260925 02:54:07.625061 19013 ts_tablet_manager.cc:1939] T 1673e73b396c494fbb4b2d0e976a5ec8 P 57b40c64bfbf403cbaa1d2b4bc30d271: Deleting consensus metadata
I20260925 02:54:07.625892 18499 catalog_manager.cc:4841] TS 6be9db95a79f4e90baccabb7428c9451 (127.2.172.67:38617): tablet 1673e73b396c494fbb4b2d0e976a5ec8 (table test-table [id=26e2f5b133b549a4a6705aa47ca9644a]) successfully deleted
I20260925 02:54:07.626950 18497 catalog_manager.cc:4841] TS 57b40c64bfbf403cbaa1d2b4bc30d271 (127.2.172.66:37741): tablet 1673e73b396c494fbb4b2d0e976a5ec8 (table test-table [id=26e2f5b133b549a4a6705aa47ca9644a]) successfully deleted
Sep 25 02:54:07 dist-test-slave-g1xh krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:54:07.634439 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.598311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.2.172.65:36243)
0925 02:54:07.598498 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:07.598549 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:07.610323 (+ 11774us) server_negotiation.cc:407] Connection header received
0925 02:54:07.610674 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:07.610680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:07.610766 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:07.610846 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:07.613822 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.614999 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.617204 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.617906 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.618006 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.628528 (+ 10522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:07.628543 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:07.628557 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:07.628593 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:07.631452 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:07.632308 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:07.632315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:07.632317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:07.632452 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:07.632843 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:07.632848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:07.632851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:07.633202 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:07.633528 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:07.633846 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 02:54:07.634071 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:54:07.635095 18697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:54:07.727252 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 18574
I20260925 02:54:07.753340 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 18708
Sep 25 02:54:07 dist-test-slave-g1xh krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:54:07.769469 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.754186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.2.172.66:37741)
0925 02:54:07.754396 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:07.754402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:07.754422 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:54:07.754546 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:07.754551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:07.754627 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:07.754730 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:07.756314 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.757181 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.758513 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.758921 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.759052 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.763314 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:07.763331 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:07.763354 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:07.763395 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:07.766388 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:07.767160 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:07.767167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:07.767170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:07.767319 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:07.767785 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:07.767790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:07.767793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:07.768191 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:07.768501 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:07.768837 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 02:54:07.769068 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:54:07.770160 18831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:54:07.883118 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 18708
I20260925 02:54:07.901746 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 18842
Sep 25 02:54:07 dist-test-slave-g1xh krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:54:07.927893 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:07.902581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.2.172.67:38617)
0925 02:54:07.902760 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:07.902764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:07.902855 (+ 91us) server_negotiation.cc:407] Connection header received
0925 02:54:07.903041 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:07.903045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:07.903111 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:07.903208 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:07.905015 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.906007 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.908026 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:07.908460 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:07.908559 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:07.920921 (+ 12362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:07.920938 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:07.920955 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:07.921002 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:07.924479 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:07.925481 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:07.925489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:07.925491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:07.925637 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:07.926100 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:07.926103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:07.926104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:07.926450 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:07.926739 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:07.927169 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 02:54:07.927370 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:54:07.928805 18965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:54:08.055397 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 18842
I20260925 02:54:08.073609 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 18481
I20260925 02:54:08.074792 18543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:54:08.186225 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 18481
I20260925 02:54:28.209254 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:33377
--webserver_interface=127.2.172.126
--webserver_port=39331
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:54:28.549016 19020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:28.549296 19020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:28.549494 19020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:28.557503 19020 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:54:28.557606 19020 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:54:28.557713 19020 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:54:28.557791 19020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:54:28.557893 19020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:28.557999 19020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:28.558081 19020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:54:28.558207 19020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:54:28.568516 19020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:33377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=39331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:54:28.569849 19020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:28.571214 19020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:28.594139 19026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:28.594170 19029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:28.594080 19027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:28.594965 19020 server_base.cc:1034] running on GCE node
I20260925 02:54:28.596015 19020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:28.597689 19020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:28.598986 19020 hybrid_clock.cc:648] HybridClock initialized: now 1790304868598908 us; error 46 us; skew 500 ppm
I20260925 02:54:28.599437 19020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:54:28 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304868, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:28.608446 19020 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:54:28.610378 19020 webserver.cc:460] Webserver started at http://127.2.172.126:39331/ using document root <none> and password file <none>
I20260925 02:54:28.611200 19020 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:28.611366 19020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:28.617550 19020 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:54:28.620594 19036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:28.621946 19020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:54:28.622135 19020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "b490732a5d124f1f999c40953816c7c6"
format_stamp: "Formatted at 2026-09-25 02:54:05 on dist-test-slave-g1xh"
server_key: "c7e45aebaf8a4bef6686efe69ba7ef07"
server_key_iv: "e9a1ec1f914d560a7202a52731c6e2ed"
server_key_version: "encryptionkey@0"
I20260925 02:54:28.623188 19020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:28.643460 19020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:28.648231 19020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:28.648597 19020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:28.668776 19020 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:33377
I20260925 02:54:28.668817 19087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:33377 every 8 connection(s)
I20260925 02:54:28.670387 19020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:54:28.676998 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 19020
I20260925 02:54:28.678076 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:36243
--local_ip_for_outbound_sockets=127.2.172.65
--tserver_master_addrs=127.2.172.126:33377
--webserver_port=36425
--webserver_interface=127.2.172.65
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:54:28.678514 19088 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:54:28.684017 19088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Bootstrap starting.
I20260925 02:54:28.706259 19088 log.cc:824] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:28.718772 19088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:54:28.719381 19088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Bootstrap complete.
I20260925 02:54:28.728884 19088 raft_consensus.cc:348] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } }
I20260925 02:54:28.729801 19088 raft_consensus.cc:729] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b490732a5d124f1f999c40953816c7c6, State: Initialized, Role: FOLLOWER
I20260925 02:54:28.730453 19088 consensus_queue.cc:260] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } }
I20260925 02:54:28.730695 19088 raft_consensus.cc:388] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:54:28.730814 19088 raft_consensus.cc:482] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:54:28.730999 19088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:54:28.734925 19088 raft_consensus.cc:504] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } }
I20260925 02:54:28.735494 19088 leader_election.cc:302] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b490732a5d124f1f999c40953816c7c6; no voters:
I20260925 02:54:28.736229 19088 leader_election.cc:288] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:54:28.736371 19092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:54:28.737635 19092 raft_consensus.cc:686] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 2 LEADER]: Becoming Leader. State: Replica: b490732a5d124f1f999c40953816c7c6, State: Running, Role: LEADER
I20260925 02:54:28.738317 19092 consensus_queue.cc:237] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } }
I20260925 02:54:28.738703 19088 sys_catalog.cc:564] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:54:28.740967 19094 sys_catalog.cc:455] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b490732a5d124f1f999c40953816c7c6. Latest consensus state: current_term: 2 leader_uuid: "b490732a5d124f1f999c40953816c7c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } } }
I20260925 02:54:28.741397 19094 sys_catalog.cc:458] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:54:28.742004 19093 sys_catalog.cc:455] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b490732a5d124f1f999c40953816c7c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } } }
I20260925 02:54:28.742354 19093 sys_catalog.cc:458] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:54:28.748867 19098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:54:28.757407 19098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:54:28.759027 19098 catalog_manager.cc:1259] Loaded cluster ID: 8a0da812999743db95af613aac785841
I20260925 02:54:28.759215 19098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:54:28.763247 19098 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:54:28.765009 19098 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Loaded TSK: 0
I20260925 02:54:28.790040 19110 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:54:28.797133 19098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Generated new TSK 1
I20260925 02:54:28.797549 19098 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:54:29.041612 19090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:29.041916 19090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:29.042073 19090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:29.050658 19090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:29.050817 19090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:29.051007 19090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:54:29.061185 19090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:36243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=36425
--tserver_master_addrs=127.2.172.126:33377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:54:29.062588 19090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:29.063979 19090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:29.079419 19116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:29.079752 19119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:29.079442 19117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:29.081151 19090 server_base.cc:1034] running on GCE node
I20260925 02:54:29.081858 19090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:29.083004 19090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:29.084255 19090 hybrid_clock.cc:648] HybridClock initialized: now 1790304869084188 us; error 59 us; skew 500 ppm
I20260925 02:54:29.084638 19090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:54:29 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304869, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:29.088066 19090 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:54:29.089722 19090 webserver.cc:460] Webserver started at http://127.2.172.65:36425/ using document root <none> and password file <none>
I20260925 02:54:29.090477 19090 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:29.090667 19090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:29.096987 19090 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:54:29.100455 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:29.101979 19090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:54:29.102190 19090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "a7694f1cbccf47178140c221f9414f22"
format_stamp: "Formatted at 2026-09-25 02:54:06 on dist-test-slave-g1xh"
server_key: "06b474b1498d68f7de6c2a207a17935f"
server_key_iv: "737fa3fa2e545574e2666ac675f4153e"
server_key_version: "encryptionkey@0"
I20260925 02:54:29.103253 19090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:29.128715 19090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:29.133359 19090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:29.133661 19090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:29.135018 19090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:54:29.136734 19090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:54:29.136881 19090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:29.137041 19090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:54:29.137140 19090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:29.173872 19090 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:36243
I20260925 02:54:29.173933 19238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:36243 every 8 connection(s)
I20260925 02:54:29.175765 19090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:54:29.177768 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 19090
I20260925 02:54:29.178408 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:37741
--local_ip_for_outbound_sockets=127.2.172.66
--tserver_master_addrs=127.2.172.126:33377
--webserver_port=33799
--webserver_interface=127.2.172.66
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:54:29 dist-test-slave-g1xh krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304869, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:54:29.213264 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:29.186137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:53385)
0925 02:54:29.187130 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:29.187180 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:29.189371 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:29.191039 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:29.191047 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:29.191723 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:54:29.192408 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:29.192430 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:29.195257 (+ 2827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:29.195263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:29.196758 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:29.196774 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:29.198556 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:29.198562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:29.198771 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:29.199724 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:54:29.199751 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:54:29.202552 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:54:29.208390 (+ 5838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:29.208399 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:29.208414 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:29.209094 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:29.209691 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:29.209695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:29.209698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:29.209928 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:29.210885 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:54:29.210892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:54:29.211325 (+ 433us) client_negotiation.cc:769] Sending connection context
0925 02:54:29.211689 (+ 364us) client_negotiation.cc:240] Negotiation successful
0925 02:54:29.212144 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":416,"threads_started":1}
I20260925 02:54:29.213536 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:29.185825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:53385 (local address 127.2.172.126:33377)
0925 02:54:29.186596 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:29.186618 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:29.187351 (+ 733us) server_negotiation.cc:407] Connection header received
0925 02:54:29.190001 (+ 2650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:29.190021 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:29.190492 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:29.190881 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:29.192747 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:29.194983 (+ 2236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:29.196966 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:29.198153 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:29.198372 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:29.202990 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:29.203008 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:29.203036 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:29.203111 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:29.208118 (+ 5007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:29.209285 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:29.209295 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:29.209304 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:29.209535 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:29.210053 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:29.210058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:29.210062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:29.210719 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:29.210918 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:29.211914 (+ 996us) server_negotiation.cc:299] Negotiation successful
0925 02:54:29.212368 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":323,"threads_started":1}
I20260925 02:54:29.216728 19239 heartbeater.cc:347] Connected to a master server at 127.2.172.126:33377
I20260925 02:54:29.217080 19239 heartbeater.cc:464] Registering TS with master...
I20260925 02:54:29.217893 19239 heartbeater.cc:511] Master 127.2.172.126:33377 requested a full tablet report, sending...
I20260925 02:54:29.220392 19053 ts_manager.cc:194] Registered new tserver with Master: a7694f1cbccf47178140c221f9414f22 (127.2.172.65:36243)
I20260925 02:54:29.222026 19053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:53385
W20260925 02:54:29.534718 19241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:29.535105 19241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:29.535238 19241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:29.542941 19241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:29.543062 19241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:29.543188 19241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:54:29.557760 19241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:37741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=33799
--tserver_master_addrs=127.2.172.126:33377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:54:29.559069 19241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:29.560468 19241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:29.573782 19248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:29.573748 19249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:29.573719 19251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:29.574452 19241 server_base.cc:1034] running on GCE node
I20260925 02:54:29.575165 19241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:29.576261 19241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:29.577487 19241 hybrid_clock.cc:648] HybridClock initialized: now 1790304869577418 us; error 60 us; skew 500 ppm
I20260925 02:54:29.577868 19241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:54:29 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304869, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:29.581207 19241 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:54:29.582723 19241 webserver.cc:460] Webserver started at http://127.2.172.66:33799/ using document root <none> and password file <none>
I20260925 02:54:29.583374 19241 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:29.583489 19241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:29.589193 19241 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 02:54:29.592046 19258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:29.593609 19241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 02:54:29.593819 19241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "57b40c64bfbf403cbaa1d2b4bc30d271"
format_stamp: "Formatted at 2026-09-25 02:54:06 on dist-test-slave-g1xh"
server_key: "1d1a859448e3c39abb2203da6a690119"
server_key_iv: "78906020cd91e52628baf7841cd847e5"
server_key_version: "encryptionkey@0"
I20260925 02:54:29.595021 19241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:29.615355 19241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:29.620231 19241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:29.620540 19241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:29.621609 19241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:54:29.623240 19241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:54:29.623409 19241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:29.623581 19241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:54:29.623687 19241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 02:54:29.658829 19241 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:37741
I20260925 02:54:29.658919 19370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:37741 every 8 connection(s)
I20260925 02:54:29.660342 19241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:54:29.668355 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 19241
I20260925 02:54:29.669154 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:38617
--local_ip_for_outbound_sockets=127.2.172.67
--tserver_master_addrs=127.2.172.126:33377
--webserver_port=43015
--webserver_interface=127.2.172.67
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:54:29 dist-test-slave-g1xh krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304869, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:54:29.684409 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:29.663095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:46753 (local address 127.2.172.126:33377)
0925 02:54:29.663337 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:29.663344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:29.664593 (+ 1249us) server_negotiation.cc:407] Connection header received
0925 02:54:29.666379 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:29.666384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:29.666456 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:29.666546 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:29.668357 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:29.669483 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:29.671919 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:29.672727 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:29.672881 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:29.677644 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:29.677654 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:29.677661 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:29.677712 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:29.681150 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:29.681922 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:29.681927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:29.681931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:29.682102 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:29.682557 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:29.682562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:29.682565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:29.682855 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:29.683011 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:29.683792 (+ 781us) server_negotiation.cc:299] Negotiation successful
0925 02:54:29.684033 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 02:54:29.685400 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:29.663668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:46753)
0925 02:54:29.664407 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:29.664452 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:29.666099 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:29.666775 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:29.666783 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:29.667334 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:54:29.668138 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:29.668166 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:29.669680 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:29.669685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:29.671725 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:29.671743 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:29.672999 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:29.673008 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:29.673287 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:29.674074 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:54:29.674121 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:54:29.677294 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:54:29.681305 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:29.681315 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:29.681340 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:29.681781 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:29.682223 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:29.682228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:29.682232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:29.682431 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:29.682988 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:54:29.682996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:54:29.683349 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 02:54:29.683703 (+ 354us) client_negotiation.cc:240] Negotiation successful
0925 02:54:29.684262 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":291,"threads_started":1}
I20260925 02:54:29.687153 19371 heartbeater.cc:347] Connected to a master server at 127.2.172.126:33377
I20260925 02:54:29.687480 19371 heartbeater.cc:464] Registering TS with master...
I20260925 02:54:29.688261 19371 heartbeater.cc:511] Master 127.2.172.126:33377 requested a full tablet report, sending...
I20260925 02:54:29.689724 19053 ts_manager.cc:194] Registered new tserver with Master: 57b40c64bfbf403cbaa1d2b4bc30d271 (127.2.172.66:37741)
I20260925 02:54:29.690601 19053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:46753
W20260925 02:54:30.009822 19374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:30.010104 19374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:30.010254 19374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:30.017590 19374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:30.017714 19374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:30.017891 19374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:54:30.027828 19374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:38617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=43015
--tserver_master_addrs=127.2.172.126:33377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:54:30.029127 19374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:30.030419 19374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:30.044131 19379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:30.044178 19380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:30.044164 19382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:30.044983 19374 server_base.cc:1034] running on GCE node
I20260925 02:54:30.045571 19374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:30.046496 19374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:30.047689 19374 hybrid_clock.cc:648] HybridClock initialized: now 1790304870047619 us; error 28 us; skew 500 ppm
I20260925 02:54:30.048022 19374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:54:30 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304870, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:30.051280 19374 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:54:30.052736 19374 webserver.cc:460] Webserver started at http://127.2.172.67:43015/ using document root <none> and password file <none>
I20260925 02:54:30.053395 19374 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:30.053543 19374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:30.058616 19374 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 02:54:30.061299 19389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:30.062493 19374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:54:30.062678 19374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "6be9db95a79f4e90baccabb7428c9451"
format_stamp: "Formatted at 2026-09-25 02:54:07 on dist-test-slave-g1xh"
server_key: "f9088126d5d5d908841e51e5fa1f03e0"
server_key_iv: "a331b8599f456943abf25fef8502a7ff"
server_key_version: "encryptionkey@0"
I20260925 02:54:30.063650 19374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:30.080278 19374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:30.084496 19374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:30.084785 19374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:30.085755 19374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:54:30.087051 19374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:54:30.087169 19374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:30.087296 19374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:54:30.087386 19374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:30.121345 19374 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:38617
I20260925 02:54:30.121428 19501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:38617 every 8 connection(s)
I20260925 02:54:30.122802 19374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:54:30.131007 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 19374
Sep 25 02:54:30 dist-test-slave-g1xh krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304870, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:54:30.144110 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.125256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:40423 (local address 127.2.172.126:33377)
0925 02:54:30.125474 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.125480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.126603 (+ 1123us) server_negotiation.cc:407] Connection header received
0925 02:54:30.128586 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.128590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.128671 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.128773 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:30.130295 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.131336 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.133492 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.133934 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.134070 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.137627 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:30.137633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:30.137636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:30.137672 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:30.140487 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:30.141176 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:30.141186 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:30.141188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:30.141309 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:30.141956 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:30.141960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:30.141962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:30.142316 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:30.142473 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:30.143397 (+ 924us) server_negotiation.cc:299] Negotiation successful
0925 02:54:30.143830 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:54:30.145403 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.125740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:40423)
0925 02:54:30.126439 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.126491 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:30.128298 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:30.129021 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:30.129029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:30.129553 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:54:30.130127 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:30.130146 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.131499 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.131503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:30.133348 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:30.133361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.134170 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.134178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:30.134401 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.135058 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:54:30.135080 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:54:30.137415 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:54:30.140621 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:30.140629 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:30.140646 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:30.141046 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:30.141445 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:30.141448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:30.141451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:30.141800 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:30.142486 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:54:30.142493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:54:30.142975 (+ 482us) client_negotiation.cc:769] Sending connection context
0925 02:54:30.143306 (+ 331us) client_negotiation.cc:240] Negotiation successful
0925 02:54:30.143909 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":261,"threads_started":1}
I20260925 02:54:30.147114 19502 heartbeater.cc:347] Connected to a master server at 127.2.172.126:33377
I20260925 02:54:30.147258 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.132047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34820 (local address 127.2.172.126:33377)
0925 02:54:30.132512 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.132516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.132957 (+ 441us) server_negotiation.cc:407] Connection header received
0925 02:54:30.133148 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.133151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.133207 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.133316 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:30.134886 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.135655 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.137274 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.137699 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.137830 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.140878 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:30.140883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:30.140886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:30.140914 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:30.144195 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:30.145055 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:30.145060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:30.145063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:30.145189 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:30.145775 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:30.145778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:30.145780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:30.146035 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:30.146316 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:30.146628 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 02:54:30.146846 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":267,"threads_started":1}
I20260925 02:54:30.147464 19502 heartbeater.cc:464] Registering TS with master...
I20260925 02:54:30.148324 19502 heartbeater.cc:511] Master 127.2.172.126:33377 requested a full tablet report, sending...
I20260925 02:54:30.149909 19053 ts_manager.cc:194] Registered new tserver with Master: 6be9db95a79f4e90baccabb7428c9451 (127.2.172.67:38617)
I20260925 02:54:30.150794 19053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:40423
I20260925 02:54:30.161872 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:54:30.168861 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.162899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.2.172.126:33377)
0925 02:54:30.163121 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.163130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.163505 (+ 375us) server_negotiation.cc:407] Connection header received
0925 02:54:30.163767 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.163772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.163938 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.164083 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:54:30.164464 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.165223 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.166817 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.167288 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.167383 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.167926 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:54:30.168205 (+ 279us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 02:54:30.168499 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34832: token expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 02:54:30.169003 19505 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34832: token expired
W20260925 02:54:30.169471 18976 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.172.126:33377)
I20260925 02:54:30.182741 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.169962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.2.172.126:33377)
0925 02:54:30.170134 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.170141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.170203 (+ 62us) server_negotiation.cc:407] Connection header received
0925 02:54:30.170353 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.170359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.170427 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.170529 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:30.171993 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.172756 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.174301 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.174719 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.174884 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.177540 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:30.177546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:30.177550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:30.177589 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:30.180017 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:30.180724 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:30.180730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:30.180734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:30.180890 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:30.181273 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:30.181276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:30.181278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:30.181588 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:30.181840 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:30.182187 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 02:54:30.182414 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:54:30.186216 19053 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:54:30.193498 19053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:54:30.225584 19239 heartbeater.cc:503] Master 127.2.172.126:33377 was elected leader, sending a full tablet report...
I20260925 02:54:30.228641 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.205682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:37741 (local address 127.0.0.1:59222)
0925 02:54:30.219578 (+ 13896us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.219600 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:30.219712 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:30.220635 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:30.220641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:30.220669 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:30.221038 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:30.221051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.223050 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.223057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:30.225419 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:30.225436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.227358 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.227365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:30.227623 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.227685 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 02:54:30.227804 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 02:54:30.228035 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13727,"mutex_wait_us":6,"thread_start_us":12413,"threads_started":1}
I20260925 02:54:30.228871 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.205710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.2.172.67:38617)
0925 02:54:30.216962 (+ 11252us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.216976 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.218719 (+ 1743us) server_negotiation.cc:407] Connection header received
0925 02:54:30.219153 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.219164 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.219456 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.219687 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:30.220763 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.222427 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.225005 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.226346 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.227968 (+ 1622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.228220 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:30.228329 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:54:30.228507 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10993,"thread_start_us":10787,"threads_started":1}
I20260925 02:54:30.228681 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.205281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:38617 (local address 127.0.0.1:46792)
0925 02:54:30.218529 (+ 13248us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.218588 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:30.218893 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:30.220014 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:30.220021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:30.220051 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:30.220538 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:30.220555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.222574 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.222580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:30.224846 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:30.224858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.227535 (+ 2677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.227542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:30.227715 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.227742 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:54:30.227801 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:54:30.228035 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12991,"mutex_wait_us":95,"thread_start_us":13984,"threads_started":1}
I20260925 02:54:30.230402 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.205954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.2.172.66:37741)
0925 02:54:30.217930 (+ 11976us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.217941 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.219842 (+ 1901us) server_negotiation.cc:407] Connection header received
0925 02:54:30.219960 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.219968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.220184 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.220379 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:30.221252 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.222864 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.225624 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.227081 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.229177 (+ 2096us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.229372 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:30.229475 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 02:54:30.229658 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11836,"thread_start_us":11722,"threads_started":1}
I20260925 02:54:30.230515 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.205997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:36243 (local address 127.0.0.1:54504)
0925 02:54:30.222082 (+ 16085us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.222098 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:30.222213 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:30.223084 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:30.223091 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:30.223123 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:30.223663 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:30.223676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.225609 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.225614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:30.227887 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:30.227901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.229787 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.229794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:30.229951 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.229985 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:54:30.230080 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 02:54:30.230203 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15974,"thread_start_us":15826,"threads_started":1}
I20260925 02:54:30.230644 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.206567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.2.172.65:36243)
0925 02:54:30.221676 (+ 15109us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.221685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.222209 (+ 524us) server_negotiation.cc:407] Connection header received
0925 02:54:30.222354 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.222361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.222550 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.222750 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:30.223875 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.225442 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.228066 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.229566 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.229704 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.229931 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:30.230168 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 02:54:30.230322 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14928,"thread_start_us":14796,"threads_started":1}
I20260925 02:54:30.233640 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 75cfa515d055467cbfbff693af49fd92 (DEFAULT_TABLE table=test-table [id=2efe96c657a74956ab4a7261ebaada0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:54:30.233448 19437 tablet_service.cc:1467] Processing CreateTablet for tablet 75cfa515d055467cbfbff693af49fd92 (DEFAULT_TABLE table=test-table [id=2efe96c657a74956ab4a7261ebaada0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:54:30.234146 19306 tablet_service.cc:1467] Processing CreateTablet for tablet 75cfa515d055467cbfbff693af49fd92 (DEFAULT_TABLE table=test-table [id=2efe96c657a74956ab4a7261ebaada0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:54:30.234970 19174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75cfa515d055467cbfbff693af49fd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:54:30.234999 19437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75cfa515d055467cbfbff693af49fd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:54:30.235361 19306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75cfa515d055467cbfbff693af49fd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:54:30.247915 19518 tablet_bootstrap.cc:492] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: Bootstrap starting.
I20260925 02:54:30.252328 19520 tablet_bootstrap.cc:492] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451: Bootstrap starting.
I20260925 02:54:30.252774 19518 tablet_bootstrap.cc:654] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: Neither blocks nor log segments found. Creating new log.
I20260925 02:54:30.254205 19518 log.cc:824] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:30.255782 19519 tablet_bootstrap.cc:492] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271: Bootstrap starting.
I20260925 02:54:30.256100 19520 tablet_bootstrap.cc:654] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451: Neither blocks nor log segments found. Creating new log.
I20260925 02:54:30.257469 19520 log.cc:824] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:30.257879 19518 tablet_bootstrap.cc:492] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: No bootstrap required, opened a new log
I20260925 02:54:30.258229 19518 ts_tablet_manager.cc:1397] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 02:54:30.260473 19519 tablet_bootstrap.cc:654] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271: Neither blocks nor log segments found. Creating new log.
I20260925 02:54:30.262092 19519 log.cc:824] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:30.264772 19520 tablet_bootstrap.cc:492] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451: No bootstrap required, opened a new log
I20260925 02:54:30.265053 19520 ts_tablet_manager.cc:1397] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260925 02:54:30.265285 19519 tablet_bootstrap.cc:492] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271: No bootstrap required, opened a new log
I20260925 02:54:30.265569 19519 ts_tablet_manager.cc:1397] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 02:54:30.267441 19518 raft_consensus.cc:348] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } } peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } }
I20260925 02:54:30.267977 19518 raft_consensus.cc:374] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:54:30.268123 19518 raft_consensus.cc:729] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7694f1cbccf47178140c221f9414f22, State: Initialized, Role: FOLLOWER
I20260925 02:54:30.268777 19518 consensus_queue.cc:260] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } } peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } }
I20260925 02:54:30.271394 19518 ts_tablet_manager.cc:1428] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 02:54:30.276324 19520 raft_consensus.cc:348] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } } peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } }
I20260925 02:54:30.276950 19520 raft_consensus.cc:374] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:54:30.276786 19519 raft_consensus.cc:348] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } } peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } }
I20260925 02:54:30.277092 19520 raft_consensus.cc:729] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be9db95a79f4e90baccabb7428c9451, State: Initialized, Role: FOLLOWER
I20260925 02:54:30.277298 19519 raft_consensus.cc:374] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:54:30.277451 19519 raft_consensus.cc:729] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b40c64bfbf403cbaa1d2b4bc30d271, State: Initialized, Role: FOLLOWER
I20260925 02:54:30.277773 19520 consensus_queue.cc:260] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } } peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } }
I20260925 02:54:30.278023 19519 consensus_queue.cc:260] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } } peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } }
I20260925 02:54:30.279265 19502 heartbeater.cc:503] Master 127.2.172.126:33377 was elected leader, sending a full tablet report...
I20260925 02:54:30.281301 19371 heartbeater.cc:503] Master 127.2.172.126:33377 was elected leader, sending a full tablet report...
I20260925 02:54:30.281538 19520 ts_tablet_manager.cc:1428] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260925 02:54:30.281952 19519 ts_tablet_manager.cc:1428] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271: Time spent starting tablet: real 0.016s user 0.011s sys 0.006s
W20260925 02:54:30.374408 19503 tablet.cc:2369] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:54:30.412364 19372 tablet.cc:2369] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:54:30.428361 19240 tablet.cc:2369] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:54:30.481678 19524 raft_consensus.cc:482] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:54:30.481966 19524 raft_consensus.cc:504] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } } peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } }
I20260925 02:54:30.483843 19524 leader_election.cc:288] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6be9db95a79f4e90baccabb7428c9451 (127.2.172.67:38617), 57b40c64bfbf403cbaa1d2b4bc30d271 (127.2.172.66:37741)
I20260925 02:54:30.494046 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.484399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:33117 (local address 127.2.172.67:38617)
0925 02:54:30.484708 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.484714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.485258 (+ 544us) server_negotiation.cc:407] Connection header received
0925 02:54:30.485441 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.485447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.485528 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.485639 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:30.486574 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.487726 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.491230 (+ 3504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.492559 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.492745 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.492826 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:30.493617 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 02:54:30.493774 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260925 02:54:30.494046 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.484413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:52049 (local address 127.2.172.66:37741)
0925 02:54:30.484652 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.484660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.485260 (+ 600us) server_negotiation.cc:407] Connection header received
0925 02:54:30.485442 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.485448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.485528 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.485639 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:30.486574 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.487726 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.491255 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.492558 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.492747 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.492826 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:30.493617 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 02:54:30.493766 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:54:30.494336 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.484118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:38617 (local address 127.2.172.65:33117)
0925 02:54:30.485061 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.485097 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:30.485270 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:30.485713 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:30.485719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:30.485754 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:30.486386 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:30.486409 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.487926 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.487933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:30.491005 (+ 3072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:30.491025 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.492872 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.492876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:30.493269 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.493425 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 02:54:30.493537 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:54:30.493779 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":267,"thread_start_us":421,"threads_started":1}
I20260925 02:54:30.494357 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.484263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:37741 (local address 127.2.172.65:52049)
0925 02:54:30.485106 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.485127 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:30.485315 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:30.485713 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:30.485719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:30.485765 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:30.486410 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:30.486419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.487926 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.487933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:30.491011 (+ 3078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:30.491025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.492864 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.492871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:30.493268 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.493425 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 02:54:30.493537 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:54:30.493779 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":638,"spinlock_wait_cycles":61056,"thread_start_us":562,"threads_started":1}
I20260925 02:54:30.495875 19457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cfa515d055467cbfbff693af49fd92" candidate_uuid: "a7694f1cbccf47178140c221f9414f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6be9db95a79f4e90baccabb7428c9451" is_pre_election: true
I20260925 02:54:30.495875 19326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cfa515d055467cbfbff693af49fd92" candidate_uuid: "a7694f1cbccf47178140c221f9414f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" is_pre_election: true
I20260925 02:54:30.496490 19326 raft_consensus.cc:2454] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7694f1cbccf47178140c221f9414f22 in term 0.
I20260925 02:54:30.496490 19457 raft_consensus.cc:2454] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7694f1cbccf47178140c221f9414f22 in term 0.
I20260925 02:54:30.497438 19129 leader_election.cc:302] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57b40c64bfbf403cbaa1d2b4bc30d271, 6be9db95a79f4e90baccabb7428c9451, a7694f1cbccf47178140c221f9414f22; no voters:
I20260925 02:54:30.498009 19524 raft_consensus.cc:2781] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:54:30.498193 19524 raft_consensus.cc:482] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:54:30.498281 19524 raft_consensus.cc:3037] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:54:30.500701 19524 raft_consensus.cc:504] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } } peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } }
I20260925 02:54:30.501348 19524 leader_election.cc:288] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [CANDIDATE]: Term 1 election: Requested vote from peers 6be9db95a79f4e90baccabb7428c9451 (127.2.172.67:38617), 57b40c64bfbf403cbaa1d2b4bc30d271 (127.2.172.66:37741)
I20260925 02:54:30.501894 19457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cfa515d055467cbfbff693af49fd92" candidate_uuid: "a7694f1cbccf47178140c221f9414f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6be9db95a79f4e90baccabb7428c9451"
I20260925 02:54:30.501899 19326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cfa515d055467cbfbff693af49fd92" candidate_uuid: "a7694f1cbccf47178140c221f9414f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271"
I20260925 02:54:30.502110 19326 raft_consensus.cc:3037] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:54:30.502120 19457 raft_consensus.cc:3037] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:54:30.504433 19457 raft_consensus.cc:2454] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7694f1cbccf47178140c221f9414f22 in term 1.
I20260925 02:54:30.504500 19326 raft_consensus.cc:2454] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7694f1cbccf47178140c221f9414f22 in term 1.
I20260925 02:54:30.504979 19129 leader_election.cc:302] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57b40c64bfbf403cbaa1d2b4bc30d271, 6be9db95a79f4e90baccabb7428c9451, a7694f1cbccf47178140c221f9414f22; no voters:
I20260925 02:54:30.505347 19524 raft_consensus.cc:2781] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:54:30.505843 19524 raft_consensus.cc:686] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 1 LEADER]: Becoming Leader. State: Replica: a7694f1cbccf47178140c221f9414f22, State: Running, Role: LEADER
I20260925 02:54:30.506428 19524 consensus_queue.cc:237] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } } peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } }
I20260925 02:54:30.511040 19052 catalog_manager.cc:5495] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7694f1cbccf47178140c221f9414f22 (127.2.172.65). New cstate: current_term: 1 leader_uuid: "a7694f1cbccf47178140c221f9414f22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7694f1cbccf47178140c221f9414f22" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 36243 } health_report { overall_health: HEALTHY } } }
I20260925 02:54:30.597769 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.592384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.2.172.65:36243)
0925 02:54:30.592604 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.592609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.592628 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:54:30.592743 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.592747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.592828 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.592957 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:54:30.593342 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.594100 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.595485 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.595966 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.596067 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.596741 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:54:30.597271 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:54:30.597388 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:30.597489 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 02:54:30.597558 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 02:54:30.606132 19326 raft_consensus.cc:1261] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 1 FOLLOWER]: Refusing update from remote peer a7694f1cbccf47178140c221f9414f22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:54:30.606139 19457 raft_consensus.cc:1261] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451 [term 1 FOLLOWER]: Refusing update from remote peer a7694f1cbccf47178140c221f9414f22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:54:30.607208 19529 consensus_queue.cc:1035] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57b40c64bfbf403cbaa1d2b4bc30d271" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:54:30.607620 19524 consensus_queue.cc:1035] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be9db95a79f4e90baccabb7428c9451" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 38617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:54:30.631367 19053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:54:30.631664 19053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:54:30.634909 19053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Sending DeleteTablet for 3 replicas of tablet 75cfa515d055467cbfbff693af49fd92
I20260925 02:54:30.636082 19437 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cfa515d055467cbfbff693af49fd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:54:30 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:46792
I20260925 02:54:30.636214 19306 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cfa515d055467cbfbff693af49fd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:54:30 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:59222
I20260925 02:54:30.636401 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cfa515d055467cbfbff693af49fd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:54:30 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:54504
I20260925 02:54:30.637352 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 19090
I20260925 02:54:30.646453 19544 tablet_replica.cc:330] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: stopping tablet replica
I20260925 02:54:30.647078 19544 raft_consensus.cc:2229] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:54:30.647990 19544 raft_consensus.cc:2258] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:54:30.649096 19543 tablet_replica.cc:330] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271: stopping tablet replica
I20260925 02:54:30.649938 19542 tablet_replica.cc:330] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451: stopping tablet replica
I20260925 02:54:30.650292 19542 raft_consensus.cc:2229] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:54:30.650820 19542 raft_consensus.cc:2258] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:54:30.653255 19544 ts_tablet_manager.cc:1905] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:54:30.653661 19543 raft_consensus.cc:2229] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:54:30.653861 19542 ts_tablet_manager.cc:1905] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:54:30.654274 19543 raft_consensus.cc:2258] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:54:30.660012 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.638310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.2.172.65:36243)
0925 02:54:30.638897 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.638903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.638920 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:54:30.638977 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.638982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.639072 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.639170 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:30.641180 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.642426 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.644483 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.645201 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.645324 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.651196 (+ 5872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:30.651210 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:30.651227 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:30.651286 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:30.655423 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:30.656466 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:30.656476 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:30.656479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:30.656813 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:30.657349 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:30.657356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:30.657365 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:30.657847 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:30.659201 (+ 1354us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:30.659379 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 02:54:30.659655 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462}
I20260925 02:54:30.660466 19543 ts_tablet_manager.cc:1905] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:54:30.664862 19234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:54:30.666253 19543 ts_tablet_manager.cc:1918] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:54:30.666471 19543 log.cc:1196] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/75cfa515d055467cbfbff693af49fd92
I20260925 02:54:30.669898 19543 ts_tablet_manager.cc:1939] T 75cfa515d055467cbfbff693af49fd92 P 57b40c64bfbf403cbaa1d2b4bc30d271: Deleting consensus metadata
I20260925 02:54:30.672627 19037 catalog_manager.cc:4841] TS 57b40c64bfbf403cbaa1d2b4bc30d271 (127.2.172.66:37741): tablet 75cfa515d055467cbfbff693af49fd92 (table test-table [id=2efe96c657a74956ab4a7261ebaada0f]) successfully deleted
I20260925 02:54:30.678226 19542 ts_tablet_manager.cc:1918] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:54:30.678578 19542 log.cc:1196] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/75cfa515d055467cbfbff693af49fd92
I20260925 02:54:30.679574 19542 ts_tablet_manager.cc:1939] T 75cfa515d055467cbfbff693af49fd92 P 6be9db95a79f4e90baccabb7428c9451: Deleting consensus metadata
I20260925 02:54:30.682003 19039 catalog_manager.cc:4841] TS 6be9db95a79f4e90baccabb7428c9451 (127.2.172.67:38617): tablet 75cfa515d055467cbfbff693af49fd92 (table test-table [id=2efe96c657a74956ab4a7261ebaada0f]) successfully deleted
I20260925 02:54:30.779975 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 19090
I20260925 02:54:30.796943 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 19241
W20260925 02:54:30.797185 19038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.172.65:36243 (error 108)
I20260925 02:54:30.799389 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.798446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:36243 (local address 127.0.0.1:54514)
0925 02:54:30.799066 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.799204 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:36243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":394,"threads_started":1}
W20260925 02:54:30.799666 19038 catalog_manager.cc:4584] TS a7694f1cbccf47178140c221f9414f22 (127.2.172.65:36243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75cfa515d055467cbfbff693af49fd92: Network error: Client connection negotiation failed: client connection to 127.2.172.65:36243: connect: Connection refused (error 111)
I20260925 02:54:30.811583 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.797777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59226 (local address 127.2.172.66:37741)
0925 02:54:30.797965 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.797971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.798075 (+ 104us) server_negotiation.cc:407] Connection header received
0925 02:54:30.798308 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.798317 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.798406 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.798515 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:30.800142 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.800847 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.802281 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.803017 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.803168 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.805949 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:30.805964 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:30.805979 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:30.806018 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:30.808937 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:30.809588 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:30.809595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:30.809597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:30.809726 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:30.810082 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:30.810085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:30.810087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:30.810495 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:30.810710 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:30.810970 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 02:54:30.811160 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:54:30.812325 19366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:54:30.853456 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.852966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:36243 (local address 127.0.0.1:54530)
0925 02:54:30.853193 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.853306 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:36243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 02:54:30.854677 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.854230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:36243 (local address 127.0.0.1:54542)
0925 02:54:30.854446 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.854519 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:36243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 02:54:30.902835 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 19241
I20260925 02:54:30.915081 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.914608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:36243 (local address 127.0.0.1:54550)
0925 02:54:30.914847 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.914933 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:36243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 02:54:30.916064 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.915720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:36243 (local address 127.0.0.1:54554)
0925 02:54:30.915864 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.915933 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:36243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 02:54:30.921869 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 19374
I20260925 02:54:30.936338 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.922565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.2.172.67:38617)
0925 02:54:30.922785 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:30.922791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:30.922813 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:54:30.922994 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:30.923004 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:30.923077 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:30.923164 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:30.924744 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.925462 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.926802 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:30.927278 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:30.927397 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:30.930058 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:30.930077 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:30.930102 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:30.930165 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:30.933326 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:30.934113 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:30.934121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:30.934124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:30.934290 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:30.934650 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:30.934654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:30.934656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:30.935052 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:30.935299 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:30.935717 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 02:54:30.935968 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:54:30.937006 19497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:54:30.997340 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.996742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:36243 (local address 127.0.0.1:54568)
0925 02:54:30.997017 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.997171 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:36243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 02:54:30.998468 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:30.998021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:36243 (local address 127.0.0.1:54574)
0925 02:54:30.998241 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:30.998329 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:36243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 02:54:31.029337 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 19374
I20260925 02:54:31.048503 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 19020
I20260925 02:54:31.049739 19083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:54:31.164361 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 19020
I20260925 02:54:51.187500 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:33377
--webserver_interface=127.2.172.126
--webserver_port=39331
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:54:51.528077 19551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:51.528471 19551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:51.528647 19551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:51.536087 19551 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:54:51.536254 19551 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:54:51.536324 19551 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:54:51.536410 19551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:54:51.536520 19551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:51.536638 19551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:51.536729 19551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:54:51.536810 19551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:54:51.547063 19551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:33377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=39331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:54:51.548681 19551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:51.550137 19551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:51.564668 19556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:51.564611 19559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:51.564692 19557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:51.566191 19551 server_base.cc:1034] running on GCE node
I20260925 02:54:51.567055 19551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:51.568825 19551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:51.570125 19551 hybrid_clock.cc:648] HybridClock initialized: now 1790304891570037 us; error 59 us; skew 500 ppm
I20260925 02:54:51.570534 19551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:54:51 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304891, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:51.582270 19551 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:54:51.584092 19551 webserver.cc:460] Webserver started at http://127.2.172.126:39331/ using document root <none> and password file <none>
I20260925 02:54:51.584816 19551 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:51.584910 19551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:51.590687 19551 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 02:54:51.593899 19566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:51.595376 19551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:54:51.595539 19551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "b490732a5d124f1f999c40953816c7c6"
format_stamp: "Formatted at 2026-09-25 02:54:05 on dist-test-slave-g1xh"
server_key: "c7e45aebaf8a4bef6686efe69ba7ef07"
server_key_iv: "e9a1ec1f914d560a7202a52731c6e2ed"
server_key_version: "encryptionkey@0"
I20260925 02:54:51.596621 19551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:51.630342 19551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:51.635120 19551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:51.635380 19551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:51.656396 19551 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:33377
I20260925 02:54:51.656428 19617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:33377 every 8 connection(s)
I20260925 02:54:51.658080 19551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:54:51.664983 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 19551
I20260925 02:54:51.665524 19618 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:54:51.665899 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:36243
--local_ip_for_outbound_sockets=127.2.172.65
--tserver_master_addrs=127.2.172.126:33377
--webserver_port=36425
--webserver_interface=127.2.172.65
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:54:51.669169 19618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Bootstrap starting.
I20260925 02:54:51.698777 19618 log.cc:824] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:51.719907 19618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:54:51.720566 19618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Bootstrap complete.
I20260925 02:54:51.733942 19618 raft_consensus.cc:348] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } }
I20260925 02:54:51.734929 19618 raft_consensus.cc:729] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b490732a5d124f1f999c40953816c7c6, State: Initialized, Role: FOLLOWER
I20260925 02:54:51.735705 19618 consensus_queue.cc:260] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } }
I20260925 02:54:51.735970 19618 raft_consensus.cc:388] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:54:51.736102 19618 raft_consensus.cc:482] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:54:51.736303 19618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:54:51.740355 19618 raft_consensus.cc:504] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } }
I20260925 02:54:51.740900 19618 leader_election.cc:302] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b490732a5d124f1f999c40953816c7c6; no voters:
I20260925 02:54:51.741571 19618 leader_election.cc:288] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:54:51.741777 19622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:54:51.742816 19622 raft_consensus.cc:686] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [term 3 LEADER]: Becoming Leader. State: Replica: b490732a5d124f1f999c40953816c7c6, State: Running, Role: LEADER
I20260925 02:54:51.743443 19622 consensus_queue.cc:237] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } }
I20260925 02:54:51.743700 19618 sys_catalog.cc:564] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:54:51.745831 19624 sys_catalog.cc:455] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b490732a5d124f1f999c40953816c7c6. Latest consensus state: current_term: 3 leader_uuid: "b490732a5d124f1f999c40953816c7c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } } }
I20260925 02:54:51.746199 19624 sys_catalog.cc:458] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:54:51.751068 19623 sys_catalog.cc:455] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b490732a5d124f1f999c40953816c7c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b490732a5d124f1f999c40953816c7c6" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33377 } } }
I20260925 02:54:51.751494 19623 sys_catalog.cc:458] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:54:51.760614 19627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:54:51.771567 19627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:54:51.773681 19627 catalog_manager.cc:1259] Loaded cluster ID: 8a0da812999743db95af613aac785841
I20260925 02:54:51.773878 19627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:54:51.779585 19627 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:54:51.781847 19627 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Loaded TSK: 0
I20260925 02:54:51.782157 19627 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b490732a5d124f1f999c40953816c7c6: Loaded TSK: 1
F20260925 02:54:51.783635 19627 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fed1534c9ad google::LogMessage::Fail() at ??:0
I20260925 02:54:51.789191 19640 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fed1534f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fed1534c4de google::LogMessage::Flush() at ??:0
@ 0x7fed1534da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fed190b07d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fed1909f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fed2ad7b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fed2ad7a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fed2ae062a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fed2ae060c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fed2ae505e9 std::function<>::operator()() at ??:0
@ 0x7fed2ad7ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fed2ad7cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fed2ae04610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fed2ae04439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fed17f98345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fed17f9b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fed17f9b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fed17f76eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fed1b2506db start_thread at ??:0
@ 0x7fed141e661f clone at ??:0
W20260925 02:54:52.118614 19620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:52.118932 19620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:52.119041 19620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:52.126801 19620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:52.126928 19620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:52.127048 19620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:54:52.138024 19620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:36243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=36425
--tserver_master_addrs=127.2.172.126:33377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:54:52.139374 19620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:52.140792 19620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:52.154945 19666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:52.154968 19669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:52.154899 19667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:52.156168 19620 server_base.cc:1034] running on GCE node
I20260925 02:54:52.156834 19620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:52.157872 19620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:52.159143 19620 hybrid_clock.cc:648] HybridClock initialized: now 1790304892159085 us; error 49 us; skew 500 ppm
I20260925 02:54:52.159518 19620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:54:52 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:52.162717 19620 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:54:52.164273 19620 webserver.cc:460] Webserver started at http://127.2.172.65:36425/ using document root <none> and password file <none>
I20260925 02:54:52.164992 19620 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:52.165135 19620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:52.170679 19620 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:54:52.173916 19676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:52.175423 19620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:54:52.175637 19620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "a7694f1cbccf47178140c221f9414f22"
format_stamp: "Formatted at 2026-09-25 02:54:06 on dist-test-slave-g1xh"
server_key: "06b474b1498d68f7de6c2a207a17935f"
server_key_iv: "737fa3fa2e545574e2666ac675f4153e"
server_key_version: "encryptionkey@0"
I20260925 02:54:52.176635 19620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:52.197762 19620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:52.202220 19620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:52.202548 19620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:52.203650 19620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:54:52.205547 19683 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260925 02:54:52.211417 19683 ts_tablet_manager.cc:1852] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 02:54:52.212581 19683 ts_tablet_manager.cc:1905] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:54:52.220744 19683 ts_tablet_manager.cc:1918] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:54:52.221021 19683 log.cc:1196] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/75cfa515d055467cbfbff693af49fd92
I20260925 02:54:52.221598 19683 ts_tablet_manager.cc:1939] T 75cfa515d055467cbfbff693af49fd92 P a7694f1cbccf47178140c221f9414f22: Deleting consensus metadata
I20260925 02:54:52.222054 19620 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 02:54:52.222198 19620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.000s sys 0.001s
I20260925 02:54:52.222487 19620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:54:52.222587 19620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:52.258070 19620 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:36243
I20260925 02:54:52.258127 19789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:36243 every 8 connection(s)
I20260925 02:54:52.259485 19620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:54:52.264535 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:52.262556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:33415)
0925 02:54:52.263294 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:52.263528 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":275,"threads_started":1}
W20260925 02:54:52.265168 19790 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:33377 (0 consecutive failures): Network error: Failed to ping master at 127.2.172.126:33377: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
I20260925 02:54:52.266022 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:52.265709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:45331)
0925 02:54:52.265850 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:52.265920 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 02:54:52.266894 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:52.266588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:36951)
0925 02:54:52.266712 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:52.266784 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 02:54:52.267184 19790 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:54:52.269555 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 19620
I20260925 02:54:52.270195 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:37741
--local_ip_for_outbound_sockets=127.2.172.66
--tserver_master_addrs=127.2.172.126:33377
--webserver_port=33799
--webserver_interface=127.2.172.66
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:54:52.619438 19793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:52.619782 19793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:52.619881 19793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:52.627049 19793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:52.627167 19793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:52.627281 19793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:54:52.637059 19793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:37741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=33799
--tserver_master_addrs=127.2.172.126:33377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:54:52.638429 19793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:52.639863 19793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:52.653424 19798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:52.653465 19801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:52.653520 19799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:52.654436 19793 server_base.cc:1034] running on GCE node
I20260925 02:54:52.655048 19793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:52.656189 19793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:52.657395 19793 hybrid_clock.cc:648] HybridClock initialized: now 1790304892657338 us; error 63 us; skew 500 ppm
I20260925 02:54:52.657727 19793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:54:52 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:52.661010 19793 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:54:52.662557 19793 webserver.cc:460] Webserver started at http://127.2.172.66:33799/ using document root <none> and password file <none>
I20260925 02:54:52.663233 19793 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:52.663337 19793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:52.669103 19793 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:54:52.672040 19808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:52.673483 19793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:54:52.673681 19793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "57b40c64bfbf403cbaa1d2b4bc30d271"
format_stamp: "Formatted at 2026-09-25 02:54:06 on dist-test-slave-g1xh"
server_key: "1d1a859448e3c39abb2203da6a690119"
server_key_iv: "78906020cd91e52628baf7841cd847e5"
server_key_version: "encryptionkey@0"
I20260925 02:54:52.674751 19793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:52.690150 19793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:52.694716 19793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:52.695008 19793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:52.696199 19793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:54:52.697604 19793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:54:52.697741 19793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:52.697870 19793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:54:52.697985 19793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:52.732867 19793 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:37741
I20260925 02:54:52.732963 19920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:37741 every 8 connection(s)
I20260925 02:54:52.734323 19793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:54:52.739105 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 19793
I20260925 02:54:52.739198 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:52.737319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:33635)
0925 02:54:52.738048 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:52.738282 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":295,"threads_started":1}
W20260925 02:54:52.739945 19921 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:33377 (0 consecutive failures): Network error: Failed to ping master at 127.2.172.126:33377: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
I20260925 02:54:52.739961 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:38617
--local_ip_for_outbound_sockets=127.2.172.67
--tserver_master_addrs=127.2.172.126:33377
--webserver_port=43015
--webserver_interface=127.2.172.67
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:54:52.740971 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:52.740490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:39849)
0925 02:54:52.740707 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:52.740808 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 02:54:52.741989 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:52.741544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:56849)
0925 02:54:52.741731 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:52.741835 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 02:54:52.742291 19921 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:54:53.105615 19924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:53.105953 19924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:53.106134 19924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:53.113608 19924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:53.113781 19924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:53.113970 19924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:54:53.124089 19924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:38617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=43015
--tserver_master_addrs=127.2.172.126:33377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:54:53.125519 19924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:53.126884 19924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:53.140424 19929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:53.140468 19932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:53.140452 19930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:53.141620 19924 server_base.cc:1034] running on GCE node
I20260925 02:54:53.142397 19924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:53.143402 19924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:53.144717 19924 hybrid_clock.cc:648] HybridClock initialized: now 1790304893144645 us; error 67 us; skew 500 ppm
I20260925 02:54:53.145097 19924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:54:53 dist-test-slave-g1xh krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304893, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:53.148558 19924 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:54:53.150076 19924 webserver.cc:460] Webserver started at http://127.2.172.67:43015/ using document root <none> and password file <none>
I20260925 02:54:53.150796 19924 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:53.150947 19924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:53.156962 19924 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:54:53.159964 19939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:53.161453 19924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:54:53.161676 19924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "6be9db95a79f4e90baccabb7428c9451"
format_stamp: "Formatted at 2026-09-25 02:54:07 on dist-test-slave-g1xh"
server_key: "f9088126d5d5d908841e51e5fa1f03e0"
server_key_iv: "a331b8599f456943abf25fef8502a7ff"
server_key_version: "encryptionkey@0"
I20260925 02:54:53.162876 19924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:53.182672 19924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:53.187142 19924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:53.187450 19924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:53.188555 19924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:54:53.189999 19924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:54:53.190096 19924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:53.190261 19924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:54:53.190361 19924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:53.222398 19924 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:38617
I20260925 02:54:53.222461 20051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:38617 every 8 connection(s)
I20260925 02:54:53.223955 19924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:54:53.227396 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 19924
I20260925 02:54:53.229406 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:53.227134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:35319)
0925 02:54:53.228166 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:53.228459 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":395,"threads_started":1}
W20260925 02:54:53.230110 20052 heartbeater.cc:644] Failed to heartbeat to 127.2.172.126:33377 (0 consecutive failures): Network error: Failed to ping master at 127.2.172.126:33377: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
I20260925 02:54:53.230990 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:53.230655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:49207)
0925 02:54:53.230809 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:53.230884 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 02:54:53.231900 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:53.231536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:39335)
0925 02:54:53.231721 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:53.231798 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260925 02:54:53.232146 20052 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:54:53.268743 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:53.267884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:54837)
0925 02:54:53.268376 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:53.268479 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":276,"threads_started":1}
I20260925 02:54:53.743731 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:53.742818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:42889)
0925 02:54:53.743302 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:53.743413 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260925 02:54:54.233563 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:54.232738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:56739)
0925 02:54:54.233191 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:54.233302 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":211,"threads_started":1}
I20260925 02:54:54.270485 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:54.269715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:51785)
0925 02:54:54.270236 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:54.270324 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":321,"threads_started":1}
I20260925 02:54:54.745375 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:54.744662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:51943)
0925 02:54:54.745118 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:54.745200 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":229,"threads_started":1}
I20260925 02:54:55.235431 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:55.234556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:40909)
0925 02:54:55.235132 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:55.235242 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":293,"threads_started":1}
I20260925 02:54:55.272327 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:55.271517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:56489)
0925 02:54:55.272060 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:55.272149 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":228,"threads_started":1}
I20260925 02:54:55.747030 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:55.746275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:53485)
0925 02:54:55.746754 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:55.746841 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
I20260925 02:54:56.237241 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:56.236492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:40847)
0925 02:54:56.236963 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:56.237040 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":283,"threads_started":1}
I20260925 02:54:56.274082 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:56.273265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:42747)
0925 02:54:56.273732 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:56.273862 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":229,"threads_started":1}
I20260925 02:54:56.748800 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:56.748020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:44337)
0925 02:54:56.748506 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:56.748595 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":249,"threads_started":1}
I20260925 02:54:57.238976 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:57.238170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:52645)
0925 02:54:57.238677 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:57.238764 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":244,"threads_started":1}
I20260925 02:54:57.276073 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:57.275099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:45655)
0925 02:54:57.275723 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:57.275841 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":306,"threads_started":1}
I20260925 02:54:57.750711 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:57.749941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:46199)
0925 02:54:57.750440 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:57.750534 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":264,"threads_started":1}
I20260925 02:54:58.241204 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:58.240307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:43241)
0925 02:54:58.240909 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:58.240996 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":285,"threads_started":1}
I20260925 02:54:58.277757 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:58.277050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:53297)
0925 02:54:58.277501 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:58.277588 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":229,"threads_started":1}
I20260925 02:54:58.752640 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:58.751837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:53939)
0925 02:54:58.752363 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:58.752473 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":246,"threads_started":1}
I20260925 02:54:59.243206 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:59.242303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:56361)
0925 02:54:59.242838 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:59.242978 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":296,"threads_started":1}
I20260925 02:54:59.279675 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:59.278733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:36177)
0925 02:54:59.279250 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:59.279358 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":239,"threads_started":1}
I20260925 02:54:59.754477 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:59.753605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:41725)
0925 02:54:59.754179 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:59.754264 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":273,"threads_started":1}
I20260925 02:55:00.245278 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:00.244343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:43391)
0925 02:55:00.244920 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:00.245013 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":288,"threads_started":1}
I20260925 02:55:00.281435 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:00.280623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:39089)
0925 02:55:00.281127 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:00.281246 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":248,"threads_started":1}
I20260925 02:55:00.756399 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:00.755647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:60239)
0925 02:55:00.756134 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:00.756228 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260925 02:55:01.247011 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:01.246320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:37331)
0925 02:55:01.246754 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:01.246834 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":224,"threads_started":1}
I20260925 02:55:01.283217 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:01.282419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:36487)
0925 02:55:01.282894 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:01.283040 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":239,"threads_started":1}
I20260925 02:55:01.758229 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:01.757448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:48047)
0925 02:55:01.757954 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:01.758045 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":246,"threads_started":1}
I20260925 02:55:02.249074 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:02.248132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:48715)
0925 02:55:02.248684 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:02.248799 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":256,"threads_started":1}
I20260925 02:55:02.284822 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:02.284145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:46021)
0925 02:55:02.284569 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:02.284649 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":215,"threads_started":1}
I20260925 02:55:02.760159 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:02.759311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:43207)
0925 02:55:02.759837 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:02.759980 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":310,"threads_started":1}
I20260925 02:55:03.250917 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:03.250161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:38867)
0925 02:55:03.250663 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:03.250754 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":277,"threads_started":1}
I20260925 02:55:03.286816 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:03.285828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:35753)
0925 02:55:03.286514 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:03.286611 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":439,"threads_started":1}
I20260925 02:55:03.761888 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:03.761189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:46559)
0925 02:55:03.761613 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:03.761703 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":214,"threads_started":1}
I20260925 02:55:04.252717 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:04.251925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:43791)
0925 02:55:04.252431 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:04.252532 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":246,"threads_started":1}
I20260925 02:55:04.288580 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:04.287826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:51049)
0925 02:55:04.288318 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:04.288419 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":286,"threads_started":1}
I20260925 02:55:04.763746 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:04.762946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:35685)
0925 02:55:04.763424 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:04.763513 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":264,"threads_started":1}
I20260925 02:55:05.254527 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:05.253791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:50955)
0925 02:55:05.254277 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:05.254364 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":257,"threads_started":1}
I20260925 02:55:05.290352 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:05.289532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:49089)
0925 02:55:05.290061 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:05.290160 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":276,"threads_started":1}
I20260925 02:55:05.765466 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:05.764689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:38787)
0925 02:55:05.765166 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:05.765268 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":226,"threads_started":1}
I20260925 02:55:06.256567 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:06.255763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:54697)
0925 02:55:06.256257 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:06.256360 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":241,"threads_started":1}
I20260925 02:55:06.292023 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:06.291239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:33261)
0925 02:55:06.291738 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:06.291826 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":211,"threads_started":1}
I20260925 02:55:06.767455 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:06.766445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:48601)
0925 02:55:06.767077 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:06.767219 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":357,"threads_started":1}
I20260925 02:55:07.258400 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:07.257634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:46707)
0925 02:55:07.258131 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:07.258216 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":272,"threads_started":1}
I20260925 02:55:07.294101 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:07.293255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:56651)
0925 02:55:07.293790 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:07.293895 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":263,"threads_started":1}
I20260925 02:55:07.769565 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:07.768677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:35741)
0925 02:55:07.769269 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:07.769359 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":324,"threads_started":1}
I20260925 02:55:08.240612 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 19620
I20260925 02:55:08.260577 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:08.259786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.67:34515)
0925 02:55:08.260276 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:08.260373 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
I20260925 02:55:08.296063 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:08.295138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.65:60903)
0925 02:55:08.295761 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:08.295874 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":243,"threads_started":1}
I20260925 02:55:08.771384 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:08.770609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33377 (local address 127.2.172.66:37287)
0925 02:55:08.771126 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:08.771224 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.126:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":252,"threads_started":1}
I20260925 02:55:08.853130 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:08.241990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.2.172.65:36243)
0925 02:55:08.242735 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:08.242759 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:08.828427 (+585668us) server_negotiation.cc:407] Connection header received
0925 02:55:08.830253 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:08.830266 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:08.830755 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:08.831077 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:08.832696 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:08.834018 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:08.835278 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:08.836071 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:08.836225 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:08.846798 (+ 10573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:08.846808 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:08.846827 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:08.846876 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:08.849838 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:08.850567 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:08.850575 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:08.850582 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:08.850767 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:08.851123 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:08.851126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:08.851129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:08.851770 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:08.852004 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:08.852437 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 02:55:08.852808 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":333,"threads_started":1}
I20260925 02:55:08.854578 19785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:08.943065 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 19620
I20260925 02:55:08.962890 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 19793
I20260925 02:55:08.982281 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:08.964079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34838 (local address 127.2.172.66:37741)
0925 02:55:08.964732 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:08.964747 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:08.964790 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:55:08.966424 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:08.966433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:08.966908 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:08.967287 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:08.969022 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:08.970312 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:08.972116 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:08.972794 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:08.972932 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:08.975800 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:08.975807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:08.975824 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:08.975870 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:08.979078 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:08.979935 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:08.979942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:08.979949 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:08.980116 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:08.980459 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:08.980462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:08.980464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:08.980988 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:08.981222 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:08.981638 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 02:55:08.981966 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":302,"threads_started":1}
I20260925 02:55:08.983735 19916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:09.068675 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 19793
I20260925 02:55:09.088126 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 19924
I20260925 02:55:09.105993 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:09.089245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.2.172.67:38617)
0925 02:55:09.089798 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:09.089813 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:09.089851 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:55:09.091497 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:09.091508 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:09.092029 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:09.092330 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:09.093738 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:09.094988 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:09.096305 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:09.097000 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:09.097135 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:09.099838 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:09.099847 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:09.099862 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:09.099906 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:09.103009 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:09.103752 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:09.103760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:09.103767 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:09.103939 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:09.104305 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:09.104309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:09.104311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:09.104829 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:09.105085 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:09.105379 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 02:55:09.105704 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":246,"threads_started":1}
I20260925 02:55:09.107446 20047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:09.191628 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 19924
2026-09-25T02:55:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65496 ms)
[----------] 30 tests from SecurityITest (312836 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:09 dist-test-slave-g1xh krb5kdc[20111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:09 dist-test-slave-g1xh krb5kdc[20111](info): set up 2 sockets
Sep 25 02:55:09 dist-test-slave-g1xh krb5kdc[20111](info): commencing operation
krb5kdc: starting...
W20260925 02:55:10.338084 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.002s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:10 dist-test-slave-g1xh krb5kdc[20111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:55:10.623062 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:44535
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:44535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:10.959806 20124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:10.960107 20124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:10.960285 20124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:10.967672 20124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:10.967835 20124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:10.967929 20124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:10.968024 20124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:10.968112 20124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:10.978323 20124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:44535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:44535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:10.979817 20124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:10.981172 20124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:10.993868 20129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:10.993817 20130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:10.993935 20132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:10.994128 20124 server_base.cc:1034] running on GCE node
I20260925 02:55:10.995189 20124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:10.996802 20124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:10.998054 20124 hybrid_clock.cc:648] HybridClock initialized: now 1790304910997982 us; error 67 us; skew 500 ppm
I20260925 02:55:10.998446 20124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:10 dist-test-slave-g1xh krb5kdc[20111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:11.001682 20124 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:55:11.003254 20124 webserver.cc:460] Webserver started at http://127.2.172.126:46533/ using document root <none> and password file <none>
I20260925 02:55:11.004077 20124 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:11.004220 20124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:11.004602 20124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:11.007764 20124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "6609e0954e7e405197befeaef1be4589"
format_stamp: "Formatted at 2026-09-25 02:55:11 on dist-test-slave-g1xh"
server_key: "19c1722ef75359de3f5602e95226725e"
server_key_iv: "9ef50948f172f24735700f37be4c83f9"
server_key_version: "encryptionkey@0"
I20260925 02:55:11.008570 20124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "6609e0954e7e405197befeaef1be4589"
format_stamp: "Formatted at 2026-09-25 02:55:11 on dist-test-slave-g1xh"
server_key: "19c1722ef75359de3f5602e95226725e"
server_key_iv: "9ef50948f172f24735700f37be4c83f9"
server_key_version: "encryptionkey@0"
I20260925 02:55:11.013952 20124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:55:11.017455 20139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.018968 20124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:55:11.019132 20124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "6609e0954e7e405197befeaef1be4589"
format_stamp: "Formatted at 2026-09-25 02:55:11 on dist-test-slave-g1xh"
server_key: "19c1722ef75359de3f5602e95226725e"
server_key_iv: "9ef50948f172f24735700f37be4c83f9"
server_key_version: "encryptionkey@0"
I20260925 02:55:11.019308 20124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:11.047713 20124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:11.051749 20124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:11.051985 20124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:11.070621 20124 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:44535
I20260925 02:55:11.070655 20190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:44535 every 8 connection(s)
I20260925 02:55:11.072031 20124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:11.075968 20191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:11.079725 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 20124
I20260925 02:55:11.079962 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:11.080852 2737 external_mini_cluster.cc:1428] Setting key 33eb5804dd7973f4157c28c3780c5874
I20260925 02:55:11.090016 20191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589: Bootstrap starting.
I20260925 02:55:11.094151 20191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:11.095647 20191 log.cc:824] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:55:11 dist-test-slave-g1xh krb5kdc[20111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:11.099023 20191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589: No bootstrap required, opened a new log
I20260925 02:55:11.105775 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:11.083056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.2.172.126:44535)
0925 02:55:11.083866 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:11.083878 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:11.083919 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:55:11.085613 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:11.085636 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:11.086052 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:11.086379 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:11.088309 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.089641 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.091463 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.092254 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.092391 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:11.097884 (+ 5493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:11.097893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:11.097908 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:11.097943 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:11.101171 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:11.102083 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:11.102089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:11.102097 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:11.102266 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:11.102682 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:11.102685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:11.102687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:11.103255 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:11.103466 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:11.104203 (+ 737us) server_negotiation.cc:299] Negotiation successful
0925 02:55:11.104619 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":408,"threads_started":1}
I20260925 02:55:11.108425 20191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6609e0954e7e405197befeaef1be4589" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44535 } }
I20260925 02:55:11.108850 20191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:11.108982 20191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6609e0954e7e405197befeaef1be4589, State: Initialized, Role: FOLLOWER
I20260925 02:55:11.109534 20191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6609e0954e7e405197befeaef1be4589" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44535 } }
I20260925 02:55:11.109786 20191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:11.109946 20191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:11.110129 20191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:11.112516 20191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6609e0954e7e405197befeaef1be4589" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44535 } }
I20260925 02:55:11.113045 20191 leader_election.cc:302] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6609e0954e7e405197befeaef1be4589; no voters:
I20260925 02:55:11.113698 20191 leader_election.cc:288] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:11.113824 20196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:11.114971 20196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [term 1 LEADER]: Becoming Leader. State: Replica: 6609e0954e7e405197befeaef1be4589, State: Running, Role: LEADER
I20260925 02:55:11.115613 20196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6609e0954e7e405197befeaef1be4589" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44535 } }
I20260925 02:55:11.116142 20191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:11.118453 20198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6609e0954e7e405197befeaef1be4589. Latest consensus state: current_term: 1 leader_uuid: "6609e0954e7e405197befeaef1be4589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6609e0954e7e405197befeaef1be4589" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44535 } } }
I20260925 02:55:11.118480 20197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6609e0954e7e405197befeaef1be4589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6609e0954e7e405197befeaef1be4589" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 44535 } } }
I20260925 02:55:11.118950 20198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:11.118950 20197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:11.120344 20202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:11.126118 20202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:11.135058 20202 catalog_manager.cc:1347] Generated new cluster ID: 146570d3073140bd83fd97fa3ca022c2
I20260925 02:55:11.135196 20202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:11.159147 20202 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:11.161188 20202 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:11.173995 20202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589: Generated new TSK 0
I20260925 02:55:11.174912 20202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:55:11.251878 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:44535
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:11.610289 20217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:11.610595 20217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:11.610798 20217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:11.619386 20217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:11.619539 20217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:11.619764 20217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:55:11.631273 20217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:44535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:11.632742 20217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:11.634155 20217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:11.648685 20222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:11.648610 20223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:11.648653 20225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:11.649900 20217 server_base.cc:1034] running on GCE node
I20260925 02:55:11.650595 20217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:11.651768 20217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:11.653038 20217 hybrid_clock.cc:648] HybridClock initialized: now 1790304911652962 us; error 67 us; skew 500 ppm
I20260925 02:55:11.653432 20217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:11 dist-test-slave-g1xh krb5kdc[20111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304911, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:11.657078 20217 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:55:11.658705 20217 webserver.cc:460] Webserver started at http://127.2.172.65:33025/ using document root <none> and password file <none>
I20260925 02:55:11.659427 20217 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:11.659535 20217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:11.659912 20217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:11.662947 20217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "d8821144f4b647d7b5061c94b862c397"
format_stamp: "Formatted at 2026-09-25 02:55:11 on dist-test-slave-g1xh"
server_key: "d6104a98bfc7d4961bfd67514a133e00"
server_key_iv: "6f7a6636d5021a20daa9b500c26fb8ae"
server_key_version: "encryptionkey@0"
I20260925 02:55:11.663812 20217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "d8821144f4b647d7b5061c94b862c397"
format_stamp: "Formatted at 2026-09-25 02:55:11 on dist-test-slave-g1xh"
server_key: "d6104a98bfc7d4961bfd67514a133e00"
server_key_iv: "6f7a6636d5021a20daa9b500c26fb8ae"
server_key_version: "encryptionkey@0"
I20260925 02:55:11.669229 20217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 02:55:11.673214 20232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.674854 20217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:11.675065 20217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "d8821144f4b647d7b5061c94b862c397"
format_stamp: "Formatted at 2026-09-25 02:55:11 on dist-test-slave-g1xh"
server_key: "d6104a98bfc7d4961bfd67514a133e00"
server_key_iv: "6f7a6636d5021a20daa9b500c26fb8ae"
server_key_version: "encryptionkey@0"
I20260925 02:55:11.675271 20217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:11.696295 20217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:11.701683 20217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:11.702066 20217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:11.703207 20217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:11.704681 20217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:11.704758 20217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.704893 20217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:11.704962 20217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.739856 20217 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:45605
I20260925 02:55:11.739904 20344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:45605 every 8 connection(s)
I20260925 02:55:11.741633 20217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:11.742784 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 20217
I20260925 02:55:11.743009 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:55:11.743731 2737 external_mini_cluster.cc:1428] Setting key fc3a60b295edfebc31d74d7b6039142a
Sep 25 02:55:11 dist-test-slave-g1xh krb5kdc[20111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304911, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:11.762501 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:11.744375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:45923 (local address 127.2.172.126:44535)
0925 02:55:11.744821 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:11.744826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:11.745985 (+ 1159us) server_negotiation.cc:407] Connection header received
0925 02:55:11.747862 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:11.747870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:11.747957 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:11.748078 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:11.749513 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.750648 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.752232 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.752966 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.753109 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:11.756925 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:11.756930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:11.756933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:11.756967 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:11.759459 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:11.760201 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:11.760207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:11.760210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:11.760331 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:11.760755 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:11.760760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:11.760763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:11.761030 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:11.761180 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:11.761887 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 02:55:11.762116 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":213,"threads_started":1}
I20260925 02:55:11.763490 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:11.745019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:44535 (local address 127.2.172.65:45923)
0925 02:55:11.745809 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:11.745861 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:11.747596 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:11.748269 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:11.748277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:11.748787 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:11.749334 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:11.749351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.750795 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.750799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:11.752093 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:11.752105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.753180 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.753184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:11.753377 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:11.754060 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:11.754086 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:11.756691 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:11.759616 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:11.759624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:11.759643 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:11.760076 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:11.760442 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:11.760449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:11.760454 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:11.760638 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:11.761168 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:11.761175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:11.761506 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 02:55:11.761763 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 02:55:11.762309 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":332,"threads_started":1}
I20260925 02:55:11.765075 20345 heartbeater.cc:347] Connected to a master server at 127.2.172.126:44535
I20260925 02:55:11.765396 20345 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:11.766134 20345 heartbeater.cc:511] Master 127.2.172.126:44535 requested a full tablet report, sending...
I20260925 02:55:11.768323 20156 ts_manager.cc:194] Registered new tserver with Master: d8821144f4b647d7b5061c94b862c397 (127.2.172.65:45605)
I20260925 02:55:11.769727 20156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:45923
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:55:11.803864 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:44535
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:12.137226 20351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:12.137524 20351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:12.137620 20351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:12.144954 20351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:12.145077 20351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:12.145190 20351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:55:12.155134 20351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:44535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:12.156443 20351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:12.157708 20351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:12.170646 20357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:12.170646 20359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:12.170950 20356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:12.171271 20351 server_base.cc:1034] running on GCE node
I20260925 02:55:12.171967 20351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:12.172962 20351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:12.174173 20351 hybrid_clock.cc:648] HybridClock initialized: now 1790304912174128 us; error 39 us; skew 500 ppm
I20260925 02:55:12.174489 20351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:12 dist-test-slave-g1xh krb5kdc[20111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304912, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:12.177587 20351 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:55:12.179030 20351 webserver.cc:460] Webserver started at http://127.2.172.66:43653/ using document root <none> and password file <none>
I20260925 02:55:12.179806 20351 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:12.179909 20351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:12.180233 20351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:12.183110 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "48200c0c7ac9488d922dad80be4aba8b"
format_stamp: "Formatted at 2026-09-25 02:55:12 on dist-test-slave-g1xh"
server_key: "525243e6bc26dc3aad19ad7a293815b6"
server_key_iv: "b30a033ead8f3965889a58d80b2eb503"
server_key_version: "encryptionkey@0"
I20260925 02:55:12.183905 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "48200c0c7ac9488d922dad80be4aba8b"
format_stamp: "Formatted at 2026-09-25 02:55:12 on dist-test-slave-g1xh"
server_key: "525243e6bc26dc3aad19ad7a293815b6"
server_key_iv: "b30a033ead8f3965889a58d80b2eb503"
server_key_version: "encryptionkey@0"
I20260925 02:55:12.188839 20351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:55:12.192239 20366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:12.193567 20351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:12.193750 20351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "48200c0c7ac9488d922dad80be4aba8b"
format_stamp: "Formatted at 2026-09-25 02:55:12 on dist-test-slave-g1xh"
server_key: "525243e6bc26dc3aad19ad7a293815b6"
server_key_iv: "b30a033ead8f3965889a58d80b2eb503"
server_key_version: "encryptionkey@0"
I20260925 02:55:12.193955 20351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:12.209605 20351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:12.213702 20351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:12.214026 20351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:12.215030 20351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:12.216547 20351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:12.216647 20351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:12.216796 20351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:12.216873 20351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:12.250716 20351 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:35967
I20260925 02:55:12.250811 20478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:35967 every 8 connection(s)
I20260925 02:55:12.252183 20351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:12.260696 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 20351
I20260925 02:55:12.260900 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:55:12.261371 2737 external_mini_cluster.cc:1428] Setting key 787869cc960cf6108733875003123f9c
Sep 25 02:55:12 dist-test-slave-g1xh krb5kdc[20111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304912, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:12.274578 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.254916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:58127 (local address 127.2.172.126:44535)
0925 02:55:12.255118 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.255122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.256489 (+ 1367us) server_negotiation.cc:407] Connection header received
0925 02:55:12.258343 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.258349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.258444 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.258570 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:12.260146 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.260914 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.262690 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.263261 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.263400 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.268077 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:12.268082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:12.268085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:12.268118 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:12.270817 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:12.271862 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:12.271866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:12.271868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:12.272012 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:12.272519 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:12.272522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:12.272524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:12.272786 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:12.272897 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.273844 (+ 947us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.274139 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 02:55:12.275257 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.255532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:44535 (local address 127.2.172.66:58127)
0925 02:55:12.256310 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:12.256357 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:12.258061 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:12.258821 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:12.258830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:12.259385 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:12.259959 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.259976 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.261097 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.261101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.262535 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.262549 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.263607 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.263617 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.263916 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.264724 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:12.264758 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:12.267807 (+ 3049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:12.270984 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:12.270996 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:12.271016 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:12.271729 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:12.272126 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:12.272133 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:12.272139 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:12.272405 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:12.272915 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:12.272927 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:12.273413 (+ 486us) client_negotiation.cc:769] Sending connection context
0925 02:55:12.273691 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 02:55:12.274096 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":269,"threads_started":1}
I20260925 02:55:12.276814 20479 heartbeater.cc:347] Connected to a master server at 127.2.172.126:44535
I20260925 02:55:12.277177 20479 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:12.278138 20479 heartbeater.cc:511] Master 127.2.172.126:44535 requested a full tablet report, sending...
I20260925 02:55:12.279883 20156 ts_manager.cc:194] Registered new tserver with Master: 48200c0c7ac9488d922dad80be4aba8b (127.2.172.66:35967)
I20260925 02:55:12.280790 20156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:58127
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:55:12.327797 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:44535
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:12.669866 20484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:12.670204 20484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:12.670377 20484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:12.677533 20484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:12.677693 20484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:12.677882 20484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:55:12.687896 20484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:44535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:12.689261 20484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:12.690629 20484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:12.704286 20490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:12.704238 20492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:12.704321 20489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:12.705072 20484 server_base.cc:1034] running on GCE node
I20260925 02:55:12.705636 20484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:12.706528 20484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:12.707722 20484 hybrid_clock.cc:648] HybridClock initialized: now 1790304912707661 us; error 55 us; skew 500 ppm
I20260925 02:55:12.708057 20484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:12 dist-test-slave-g1xh krb5kdc[20111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304912, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:12.711249 20484 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:55:12.712769 20484 webserver.cc:460] Webserver started at http://127.2.172.67:36757/ using document root <none> and password file <none>
I20260925 02:55:12.713492 20484 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:12.713637 20484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:12.713966 20484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:12.717087 20484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "77331bb8ed054669a1c4a96a53f83fe4"
format_stamp: "Formatted at 2026-09-25 02:55:12 on dist-test-slave-g1xh"
server_key: "a25ba64face83098bb8b2134026a804c"
server_key_iv: "6e0072286fd3914b6f5d1f4cf52dde3b"
server_key_version: "encryptionkey@0"
I20260925 02:55:12.717900 20484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "77331bb8ed054669a1c4a96a53f83fe4"
format_stamp: "Formatted at 2026-09-25 02:55:12 on dist-test-slave-g1xh"
server_key: "a25ba64face83098bb8b2134026a804c"
server_key_iv: "6e0072286fd3914b6f5d1f4cf52dde3b"
server_key_version: "encryptionkey@0"
I20260925 02:55:12.723512 20484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:55:12.727233 20499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:12.728663 20484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:55:12.728842 20484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "77331bb8ed054669a1c4a96a53f83fe4"
format_stamp: "Formatted at 2026-09-25 02:55:12 on dist-test-slave-g1xh"
server_key: "a25ba64face83098bb8b2134026a804c"
server_key_iv: "6e0072286fd3914b6f5d1f4cf52dde3b"
server_key_version: "encryptionkey@0"
I20260925 02:55:12.729094 20484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:12.763414 20484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:12.767541 20484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:12.767901 20484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:12.768869 20484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:12.770193 20484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:12.770267 20484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:12.770450 20484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:12.770503 20484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:12.772652 20345 heartbeater.cc:503] Master 127.2.172.126:44535 was elected leader, sending a full tablet report...
I20260925 02:55:12.805486 20484 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:35127
I20260925 02:55:12.805536 20612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:35127 every 8 connection(s)
I20260925 02:55:12.806790 20484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:55:12.815050 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 20484
I20260925 02:55:12.815282 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:55:12.815945 2737 external_mini_cluster.cc:1428] Setting key 88718c6586c21ab291a10b1e2840aa66
Sep 25 02:55:12 dist-test-slave-g1xh krb5kdc[20111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304912, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:12.827375 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.809536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:40333 (local address 127.2.172.126:44535)
0925 02:55:12.810033 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.810039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.810805 (+ 766us) server_negotiation.cc:407] Connection header received
0925 02:55:12.812741 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.812746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.812841 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.812969 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:12.814506 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.815386 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.817542 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.818156 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.818271 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.821920 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:12.821925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:12.821928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:12.821959 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:12.824664 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:12.825387 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:12.825391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:12.825393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:12.825508 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:12.825876 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:12.825880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:12.825882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:12.826167 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:12.826275 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.826919 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.827102 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":265,"threads_started":1}
I20260925 02:55:12.828378 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.809948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:44535 (local address 127.2.172.67:40333)
0925 02:55:12.810631 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:12.810678 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:12.812447 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:12.813221 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:12.813229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:12.813758 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:12.814324 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.814340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.815538 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.815582 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.817380 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.817397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.818400 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.818408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.818708 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.819507 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:12.819538 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:12.821727 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:12.824800 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:12.824807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:12.824820 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:12.825278 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:12.825598 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:12.825601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:12.825604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:12.825781 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:12.826271 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:12.826277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:12.826597 (+ 320us) client_negotiation.cc:769] Sending connection context
0925 02:55:12.826820 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 02:55:12.827233 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":268,"threads_started":1}
I20260925 02:55:12.829934 20613 heartbeater.cc:347] Connected to a master server at 127.2.172.126:44535
I20260925 02:55:12.830297 20613 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:12.831197 20613 heartbeater.cc:511] Master 127.2.172.126:44535 requested a full tablet report, sending...
I20260925 02:55:12.832703 20156 ts_manager.cc:194] Registered new tserver with Master: 77331bb8ed054669a1c4a96a53f83fe4 (127.2.172.67:35127)
I20260925 02:55:12.833608 20156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:40333
I20260925 02:55:12.844579 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:55:12.861735 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.849275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.2.172.126:44535)
0925 02:55:12.849443 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.849448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.850009 (+ 561us) server_negotiation.cc:407] Connection header received
0925 02:55:12.850149 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.850153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.850216 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.850301 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:12.851785 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.852497 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.853797 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.854189 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.854282 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.856746 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:12.856752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:12.856755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:12.856787 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:12.859399 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:12.860045 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:12.860049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:12.860051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:12.860163 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:12.860483 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:12.860486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:12.860488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:12.860828 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:12.861002 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.861327 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.861490 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:55:12.865635 20156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:55:12.872581 20156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:55:12.908556 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.884954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.2.172.67:35127)
0925 02:55:12.900068 (+ 15114us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.900080 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.900119 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:55:12.900232 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.900242 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.900460 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.900676 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:12.901691 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.903582 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.905912 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.907385 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.907514 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.907749 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.908056 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.908228 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14938,"thread_start_us":14783,"threads_started":1}
I20260925 02:55:12.908555 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.884854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:35127 (local address 127.0.0.1:51628)
0925 02:55:12.898567 (+ 13713us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:12.899838 (+ 1271us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:12.899958 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:12.901006 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:12.901011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:12.901038 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:12.901480 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.901501 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.903708 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.903714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.905730 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.905745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.907631 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.907638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.907820 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.907873 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 02:55:12.907975 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:55:12.908105 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13534,"mutex_wait_us":90,"thread_start_us":13496,"threads_started":1}
I20260925 02:55:12.908743 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.886690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:35967 (local address 127.0.0.1:36462)
0925 02:55:12.901168 (+ 14478us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:12.901187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:12.901313 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:12.901923 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:12.901927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:12.901952 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:12.902322 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.902334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.904082 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.904088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.906092 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.906109 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.907924 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.907933 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.908099 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.908127 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:55:12.908205 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:55:12.908334 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14341,"mutex_wait_us":83,"thread_start_us":14172,"threads_started":1}
I20260925 02:55:12.909020 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.884846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38264 (local address 127.2.172.65:45605)
0925 02:55:12.897493 (+ 12647us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.897509 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.899084 (+ 1575us) server_negotiation.cc:407] Connection header received
0925 02:55:12.899224 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.899232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.899493 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.899707 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:12.900743 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.902333 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.905221 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.906757 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.906900 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.907138 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.908211 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.908379 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12363,"thread_start_us":12891,"threads_started":1}
I20260925 02:55:12.909477 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.884583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45605 (local address 127.0.0.1:38264)
0925 02:55:12.898567 (+ 13984us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:12.898640 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:12.898880 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:12.900115 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:12.900126 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:12.900156 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:12.900552 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.900565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.902504 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.902509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.905040 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.905054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.907261 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.907268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.907468 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.907542 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 02:55:12.907719 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 02:55:12.907903 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13637,"mutex_wait_us":26,"thread_start_us":13972,"threads_started":1}
I20260925 02:55:12.909497 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.887297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.2.172.66:35967)
0925 02:55:12.900710 (+ 13413us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.900718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.901302 (+ 584us) server_negotiation.cc:407] Connection header received
0925 02:55:12.901496 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.901504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.901731 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.901932 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:12.902515 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.903931 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.906270 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.907404 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.907588 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.907782 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.909099 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.909274 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13245,"thread_start_us":13129,"threads_started":1}
I20260925 02:55:12.912680 20547 tablet_service.cc:1467] Processing CreateTablet for tablet d4069363f21e49f59ccd8b7917782940 (DEFAULT_TABLE table=test-table [id=4f64918ec0974213b44f7fc454548c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:12.912683 20414 tablet_service.cc:1467] Processing CreateTablet for tablet d4069363f21e49f59ccd8b7917782940 (DEFAULT_TABLE table=test-table [id=4f64918ec0974213b44f7fc454548c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:12.913973 20547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4069363f21e49f59ccd8b7917782940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:12.914075 20414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4069363f21e49f59ccd8b7917782940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:12.916079 20280 tablet_service.cc:1467] Processing CreateTablet for tablet d4069363f21e49f59ccd8b7917782940 (DEFAULT_TABLE table=test-table [id=4f64918ec0974213b44f7fc454548c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:12.917579 20280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4069363f21e49f59ccd8b7917782940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:12.925856 20632 tablet_bootstrap.cc:492] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4: Bootstrap starting.
I20260925 02:55:12.928172 20633 tablet_bootstrap.cc:492] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b: Bootstrap starting.
I20260925 02:55:12.929623 20632 tablet_bootstrap.cc:654] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:12.930958 20632 log.cc:824] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:12.931900 20634 tablet_bootstrap.cc:492] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397: Bootstrap starting.
I20260925 02:55:12.931922 20633 tablet_bootstrap.cc:654] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:12.933327 20633 log.cc:824] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:12.935480 20632 tablet_bootstrap.cc:492] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4: No bootstrap required, opened a new log
I20260925 02:55:12.935490 20634 tablet_bootstrap.cc:654] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:12.935794 20632 ts_tablet_manager.cc:1397] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260925 02:55:12.936146 20633 tablet_bootstrap.cc:492] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b: No bootstrap required, opened a new log
I20260925 02:55:12.936393 20633 ts_tablet_manager.cc:1397] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260925 02:55:12.936733 20634 log.cc:824] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:12.939776 20634 tablet_bootstrap.cc:492] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397: No bootstrap required, opened a new log
I20260925 02:55:12.940042 20634 ts_tablet_manager.cc:1397] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 02:55:12.944358 20633 raft_consensus.cc:348] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8821144f4b647d7b5061c94b862c397" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45605 } } peers { permanent_uuid: "77331bb8ed054669a1c4a96a53f83fe4" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35127 } } peers { permanent_uuid: "48200c0c7ac9488d922dad80be4aba8b" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35967 } }
I20260925 02:55:12.944728 20633 raft_consensus.cc:374] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:12.944859 20633 raft_consensus.cc:729] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48200c0c7ac9488d922dad80be4aba8b, State: Initialized, Role: FOLLOWER
I20260925 02:55:12.945470 20633 consensus_queue.cc:260] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8821144f4b647d7b5061c94b862c397" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45605 } } peers { permanent_uuid: "77331bb8ed054669a1c4a96a53f83fe4" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35127 } } peers { permanent_uuid: "48200c0c7ac9488d922dad80be4aba8b" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35967 } }
I20260925 02:55:12.946206 20632 raft_consensus.cc:348] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8821144f4b647d7b5061c94b862c397" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45605 } } peers { permanent_uuid: "77331bb8ed054669a1c4a96a53f83fe4" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35127 } } peers { permanent_uuid: "48200c0c7ac9488d922dad80be4aba8b" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35967 } }
I20260925 02:55:12.946599 20479 heartbeater.cc:503] Master 127.2.172.126:44535 was elected leader, sending a full tablet report...
I20260925 02:55:12.946686 20632 raft_consensus.cc:374] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:12.946777 20632 raft_consensus.cc:729] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77331bb8ed054669a1c4a96a53f83fe4, State: Initialized, Role: FOLLOWER
I20260925 02:55:12.947364 20633 ts_tablet_manager.cc:1428] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260925 02:55:12.947434 20632 consensus_queue.cc:260] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8821144f4b647d7b5061c94b862c397" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45605 } } peers { permanent_uuid: "77331bb8ed054669a1c4a96a53f83fe4" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35127 } } peers { permanent_uuid: "48200c0c7ac9488d922dad80be4aba8b" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35967 } }
I20260925 02:55:12.948531 20613 heartbeater.cc:503] Master 127.2.172.126:44535 was elected leader, sending a full tablet report...
I20260925 02:55:12.949194 20632 ts_tablet_manager.cc:1428] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 02:55:12.952368 20634 raft_consensus.cc:348] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8821144f4b647d7b5061c94b862c397" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45605 } } peers { permanent_uuid: "77331bb8ed054669a1c4a96a53f83fe4" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35127 } } peers { permanent_uuid: "48200c0c7ac9488d922dad80be4aba8b" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35967 } }
I20260925 02:55:12.952889 20634 raft_consensus.cc:374] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:12.953006 20634 raft_consensus.cc:729] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8821144f4b647d7b5061c94b862c397, State: Initialized, Role: FOLLOWER
I20260925 02:55:12.953567 20634 consensus_queue.cc:260] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8821144f4b647d7b5061c94b862c397" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45605 } } peers { permanent_uuid: "77331bb8ed054669a1c4a96a53f83fe4" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35127 } } peers { permanent_uuid: "48200c0c7ac9488d922dad80be4aba8b" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35967 } }
I20260925 02:55:12.954840 20634 ts_tablet_manager.cc:1428] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
W20260925 02:55:12.994190 20346 tablet.cc:2369] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:55:13.004228 20480 tablet.cc:2369] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:55:13.058629 20614 tablet.cc:2369] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:13.384793 20639 raft_consensus.cc:482] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:13.385142 20639 raft_consensus.cc:504] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8821144f4b647d7b5061c94b862c397" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45605 } } peers { permanent_uuid: "77331bb8ed054669a1c4a96a53f83fe4" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35127 } } peers { permanent_uuid: "48200c0c7ac9488d922dad80be4aba8b" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35967 } }
I20260925 02:55:13.386832 20639 leader_election.cc:288] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8821144f4b647d7b5061c94b862c397 (127.2.172.65:45605), 48200c0c7ac9488d922dad80be4aba8b (127.2.172.66:35967)
I20260925 02:55:13.395754 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:13.387380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:44977 (local address 127.2.172.66:35967)
0925 02:55:13.387666 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:13.387675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:13.388310 (+ 635us) server_negotiation.cc:407] Connection header received
0925 02:55:13.388463 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:13.388468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:13.388550 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:13.388640 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:13.389286 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.390328 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.393008 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.394446 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.394639 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:13.394719 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:13.395277 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 02:55:13.395439 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:55:13.395754 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:13.387190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:36985 (local address 127.2.172.65:45605)
0925 02:55:13.387410 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:13.387416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:13.387996 (+ 580us) server_negotiation.cc:407] Connection header received
0925 02:55:13.388223 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:13.388228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:13.388324 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:13.388444 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:13.389077 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.390165 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.393007 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.394445 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.394636 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:13.394719 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:13.395277 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 02:55:13.395424 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:55:13.396037 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:13.387297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:35967 (local address 127.2.172.67:44977)
0925 02:55:13.388195 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:13.388211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:13.388325 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:13.388672 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:13.388676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:13.388696 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:13.389142 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:13.389152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.390454 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.390459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:13.392791 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:13.392810 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.394635 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.394645 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:13.394956 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:13.395122 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 02:55:13.395222 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:55:13.395446 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":215,"thread_start_us":558,"threads_started":1}
I20260925 02:55:13.396059 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:13.387203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45605 (local address 127.2.172.67:36985)
0925 02:55:13.387865 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:13.387882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:13.388070 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:13.388500 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:13.388504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:13.388525 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:13.388891 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:13.388903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.390312 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.390316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:13.392814 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:13.392824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.394674 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.394680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:13.394952 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:13.395122 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 02:55:13.395222 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:55:13.395445 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":223,"thread_start_us":400,"threads_started":1}
I20260925 02:55:13.397174 20434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4069363f21e49f59ccd8b7917782940" candidate_uuid: "77331bb8ed054669a1c4a96a53f83fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48200c0c7ac9488d922dad80be4aba8b" is_pre_election: true
I20260925 02:55:13.397393 20300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4069363f21e49f59ccd8b7917782940" candidate_uuid: "77331bb8ed054669a1c4a96a53f83fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8821144f4b647d7b5061c94b862c397" is_pre_election: true
I20260925 02:55:13.397792 20434 raft_consensus.cc:2454] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77331bb8ed054669a1c4a96a53f83fe4 in term 0.
I20260925 02:55:13.397989 20300 raft_consensus.cc:2454] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77331bb8ed054669a1c4a96a53f83fe4 in term 0.
I20260925 02:55:13.398566 20500 leader_election.cc:302] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48200c0c7ac9488d922dad80be4aba8b, 77331bb8ed054669a1c4a96a53f83fe4; no voters:
I20260925 02:55:13.398999 20639 raft_consensus.cc:2781] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:55:13.399166 20639 raft_consensus.cc:482] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:55:13.399281 20639 raft_consensus.cc:3037] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:13.401953 20639 raft_consensus.cc:504] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8821144f4b647d7b5061c94b862c397" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45605 } } peers { permanent_uuid: "77331bb8ed054669a1c4a96a53f83fe4" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35127 } } peers { permanent_uuid: "48200c0c7ac9488d922dad80be4aba8b" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35967 } }
I20260925 02:55:13.402709 20639 leader_election.cc:288] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [CANDIDATE]: Term 1 election: Requested vote from peers d8821144f4b647d7b5061c94b862c397 (127.2.172.65:45605), 48200c0c7ac9488d922dad80be4aba8b (127.2.172.66:35967)
I20260925 02:55:13.403301 20434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4069363f21e49f59ccd8b7917782940" candidate_uuid: "77331bb8ed054669a1c4a96a53f83fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48200c0c7ac9488d922dad80be4aba8b"
I20260925 02:55:13.403301 20300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4069363f21e49f59ccd8b7917782940" candidate_uuid: "77331bb8ed054669a1c4a96a53f83fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8821144f4b647d7b5061c94b862c397"
I20260925 02:55:13.403535 20300 raft_consensus.cc:3037] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:13.403584 20434 raft_consensus.cc:3037] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:13.406893 20434 raft_consensus.cc:2454] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77331bb8ed054669a1c4a96a53f83fe4 in term 1.
I20260925 02:55:13.406893 20300 raft_consensus.cc:2454] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77331bb8ed054669a1c4a96a53f83fe4 in term 1.
I20260925 02:55:13.407358 20500 leader_election.cc:302] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48200c0c7ac9488d922dad80be4aba8b, 77331bb8ed054669a1c4a96a53f83fe4; no voters:
I20260925 02:55:13.407737 20639 raft_consensus.cc:2781] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:13.408178 20639 raft_consensus.cc:686] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 1 LEADER]: Becoming Leader. State: Replica: 77331bb8ed054669a1c4a96a53f83fe4, State: Running, Role: LEADER
I20260925 02:55:13.408730 20639 consensus_queue.cc:237] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8821144f4b647d7b5061c94b862c397" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45605 } } peers { permanent_uuid: "77331bb8ed054669a1c4a96a53f83fe4" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35127 } } peers { permanent_uuid: "48200c0c7ac9488d922dad80be4aba8b" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35967 } }
I20260925 02:55:13.413218 20156 catalog_manager.cc:5495] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77331bb8ed054669a1c4a96a53f83fe4 (127.2.172.67). New cstate: current_term: 1 leader_uuid: "77331bb8ed054669a1c4a96a53f83fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8821144f4b647d7b5061c94b862c397" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77331bb8ed054669a1c4a96a53f83fe4" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48200c0c7ac9488d922dad80be4aba8b" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35967 } health_report { overall_health: UNKNOWN } } }
I20260925 02:55:13.476889 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:13.470793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.2.172.67:35127)
0925 02:55:13.471436 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:13.471444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:13.471467 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:55:13.471632 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:13.471636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:13.471717 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:13.471894 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:55:13.472308 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.472984 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.474609 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.475008 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.475155 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:13.475676 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:55:13.476198 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:55:13.476377 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:13.476492 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 02:55:13.476579 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":413,"threads_started":1}
I20260925 02:55:13.485332 20434 raft_consensus.cc:1261] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b [term 1 FOLLOWER]: Refusing update from remote peer 77331bb8ed054669a1c4a96a53f83fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:13.485337 20300 raft_consensus.cc:1261] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397 [term 1 FOLLOWER]: Refusing update from remote peer 77331bb8ed054669a1c4a96a53f83fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:13.486403 20643 consensus_queue.cc:1035] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8821144f4b647d7b5061c94b862c397" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:13.486758 20639 consensus_queue.cc:1035] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48200c0c7ac9488d922dad80be4aba8b" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:13.499161 20651 mvcc.cc:205] Tried to move back new op lower bound from 7333088925625430016 to 7333088925325897728. Current Snapshot: MvccSnapshot[applied={T|T < 7333088925625430016}]
I20260925 02:55:13.515002 20156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:13.515285 20156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:13.519192 20156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6609e0954e7e405197befeaef1be4589: Sending DeleteTablet for 3 replicas of tablet d4069363f21e49f59ccd8b7917782940
I20260925 02:55:13.520525 20280 tablet_service.cc:1514] Processing DeleteTablet for tablet d4069363f21e49f59ccd8b7917782940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:13 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:38264
I20260925 02:55:13.520627 20547 tablet_service.cc:1514] Processing DeleteTablet for tablet d4069363f21e49f59ccd8b7917782940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:13 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:51628
I20260925 02:55:13.521087 20414 tablet_service.cc:1514] Processing DeleteTablet for tablet d4069363f21e49f59ccd8b7917782940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:13 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:36462
I20260925 02:55:13.521595 20662 tablet_replica.cc:330] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397: stopping tablet replica
I20260925 02:55:13.522305 20662 raft_consensus.cc:2229] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:13.523137 20662 raft_consensus.cc:2258] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:13.523417 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 20217
I20260925 02:55:13.524785 20663 tablet_replica.cc:330] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4: stopping tablet replica
I20260925 02:55:13.525223 20663 raft_consensus.cc:2229] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:55:13.526036 20663 raft_consensus.cc:2258] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:13.527884 20662 ts_tablet_manager.cc:1905] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:13.528275 20663 ts_tablet_manager.cc:1905] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:13.538659 20662 ts_tablet_manager.cc:1918] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:13.540442 20663 ts_tablet_manager.cc:1918] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:13.540702 20663 log.cc:1196] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/d4069363f21e49f59ccd8b7917782940
I20260925 02:55:13.541414 20663 ts_tablet_manager.cc:1939] T d4069363f21e49f59ccd8b7917782940 P 77331bb8ed054669a1c4a96a53f83fe4: Deleting consensus metadata
I20260925 02:55:13.543481 20142 catalog_manager.cc:4841] TS 77331bb8ed054669a1c4a96a53f83fe4 (127.2.172.67:35127): tablet d4069363f21e49f59ccd8b7917782940 (table test-table [id=4f64918ec0974213b44f7fc454548c41]) successfully deleted
I20260925 02:55:13.544198 20662 log.cc:1196] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/d4069363f21e49f59ccd8b7917782940
I20260925 02:55:13.544873 20662 ts_tablet_manager.cc:1939] T d4069363f21e49f59ccd8b7917782940 P d8821144f4b647d7b5061c94b862c397: Deleting consensus metadata
I20260925 02:55:13.546891 20143 catalog_manager.cc:4841] TS d8821144f4b647d7b5061c94b862c397 (127.2.172.65:45605): tablet d4069363f21e49f59ccd8b7917782940 (table test-table [id=4f64918ec0974213b44f7fc454548c41]) successfully deleted
I20260925 02:55:13.547895 20664 tablet_replica.cc:330] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b: stopping tablet replica
I20260925 02:55:13.548444 20664 raft_consensus.cc:2229] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:13.549058 20664 raft_consensus.cc:2258] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 02:55:13 dist-test-slave-g1xh krb5kdc[20111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:55:13.554281 20664 ts_tablet_manager.cc:1905] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:13.562276 20664 ts_tablet_manager.cc:1918] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:13.562573 20664 log.cc:1196] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/d4069363f21e49f59ccd8b7917782940
I20260925 02:55:13.562575 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:13.524537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.2.172.65:45605)
0925 02:55:13.524735 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:13.524742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:13.540196 (+ 15454us) server_negotiation.cc:407] Connection header received
0925 02:55:13.540562 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:13.540572 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:13.540694 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:13.540801 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:13.543272 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.544845 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.547618 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.548392 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.548512 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:13.555342 (+ 6830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:13.555362 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:13.555382 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:13.555446 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:13.559312 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:13.560344 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:13.560355 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:13.560358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:13.560509 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:13.560940 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:13.560945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:13.560949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:13.561326 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:13.561682 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:13.561912 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 02:55:13.562147 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:55:13.563110 20664 ts_tablet_manager.cc:1939] T d4069363f21e49f59ccd8b7917782940 P 48200c0c7ac9488d922dad80be4aba8b: Deleting consensus metadata
I20260925 02:55:13.563419 20340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:13.565166 20140 catalog_manager.cc:4841] TS 48200c0c7ac9488d922dad80be4aba8b (127.2.172.66:35967): tablet d4069363f21e49f59ccd8b7917782940 (table test-table [id=4f64918ec0974213b44f7fc454548c41]) successfully deleted
I20260925 02:55:13.658270 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 20217
I20260925 02:55:13.678658 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 20351
Sep 25 02:55:13 dist-test-slave-g1xh krb5kdc[20111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:55:13.696372 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:13.679544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.2.172.66:35967)
0925 02:55:13.679774 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:13.679780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:13.679803 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:55:13.679949 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:13.679955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:13.680050 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:13.680164 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:13.681531 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.682689 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.684126 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.684792 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.684926 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:13.689427 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:13.689447 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:13.689469 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:13.689518 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:13.693248 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:13.694043 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:13.694054 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:13.694058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:13.694229 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:13.694658 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:13.694663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:13.694667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:13.695152 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:13.695475 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:13.695704 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 02:55:13.695971 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:55:13.697192 20474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:13.787294 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 20351
I20260925 02:55:13.808147 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 20484
Sep 25 02:55:13 dist-test-slave-g1xh krb5kdc[20111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:55:13.831068 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:13.809007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.2.172.67:35127)
0925 02:55:13.809184 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:13.809190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:13.809411 (+ 221us) server_negotiation.cc:407] Connection header received
0925 02:55:13.809632 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:13.809637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:13.809737 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:13.809848 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:13.811951 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.813247 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.815272 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:13.816067 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:13.816207 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:13.822564 (+ 6357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:13.822582 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:13.822602 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:13.822648 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:13.827235 (+ 4587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:13.828256 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:13.828267 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:13.828273 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:13.828463 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:13.828955 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:13.828959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:13.828961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:13.829535 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:13.829966 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:13.830179 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 02:55:13.830447 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:55:13.831934 20608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:13.958344 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 20484
I20260925 02:55:13.979482 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 20124
I20260925 02:55:13.980777 20186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:14.093393 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 20124
2026-09-25T02:55:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4907 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T02:55:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:14Z Disabled control of system clock
I20260925 02:55:14.184085 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:43303
--webserver_interface=127.2.172.126
--webserver_port=0
--builtin_ntp_servers=127.2.172.84:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:43303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:55:14.525023 20673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:14.525334 20673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:14.525507 20673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:14.533070 20673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:14.533203 20673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:14.533282 20673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:14.533370 20673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:14.533452 20673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:14.543718 20673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:43303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:43303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:14.545178 20673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:14.547103 20673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:14.559588 20681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:14.559788 20679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:14.560050 20678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:14.560776 20673 server_base.cc:1034] running on GCE node
I20260925 02:55:14.561573 20673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:14.563108 20673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:14.564339 20673 hybrid_clock.cc:648] HybridClock initialized: now 1790304914564287 us; error 27 us; skew 500 ppm
I20260925 02:55:14.564718 20673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:55:14.572649 20673 webserver.cc:460] Webserver started at http://127.2.172.126:44405/ using document root <none> and password file <none>
I20260925 02:55:14.573422 20673 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:14.573532 20673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:14.573848 20673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:14.578002 20673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "8ee0a5800a224e32bce95bb0c762767b"
format_stamp: "Formatted at 2026-09-25 02:55:14 on dist-test-slave-g1xh"
server_key: "4020941e66105b6667724dd3b3b7268d"
server_key_iv: "d0c500ed9af48c474e0c684844830365"
server_key_version: "encryptionkey@0"
I20260925 02:55:14.578987 20673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "8ee0a5800a224e32bce95bb0c762767b"
format_stamp: "Formatted at 2026-09-25 02:55:14 on dist-test-slave-g1xh"
server_key: "4020941e66105b6667724dd3b3b7268d"
server_key_iv: "d0c500ed9af48c474e0c684844830365"
server_key_version: "encryptionkey@0"
I20260925 02:55:14.585064 20673 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260925 02:55:14.588994 20687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:14.590584 20673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:55:14.590816 20673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "8ee0a5800a224e32bce95bb0c762767b"
format_stamp: "Formatted at 2026-09-25 02:55:14 on dist-test-slave-g1xh"
server_key: "4020941e66105b6667724dd3b3b7268d"
server_key_iv: "d0c500ed9af48c474e0c684844830365"
server_key_version: "encryptionkey@0"
I20260925 02:55:14.591077 20673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:14.609864 20673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:14.610770 20673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:14.611152 20673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:14.630935 20738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:43303 every 8 connection(s)
I20260925 02:55:14.630929 20673 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:43303
I20260925 02:55:14.632606 20673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:14.636569 20739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:14.641933 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 20673
I20260925 02:55:14.642135 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:14.642697 2737 external_mini_cluster.cc:1428] Setting key 6a0abe344c3a714c4d5867f9999d0ca7
I20260925 02:55:14.648357 20739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b: Bootstrap starting.
I20260925 02:55:14.652153 20739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:14.653360 20739 log.cc:824] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:14.656803 20739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b: No bootstrap required, opened a new log
I20260925 02:55:14.657195 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:14.644706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.2.172.126:43303)
0925 02:55:14.645791 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:14.645803 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:14.645844 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:55:14.647675 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:14.647700 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:14.648164 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:14.648471 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:14.649103 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:14.650400 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:14.652395 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:14.653117 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:14.653260 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:14.654653 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:14.654670 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:14.654690 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:14.655340 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:14.655417 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:14.655438 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:14.655531 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:14.655856 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 02:55:14.656229 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":412,"threads_started":1}
I20260925 02:55:14.664595 20739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee0a5800a224e32bce95bb0c762767b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43303 } }
I20260925 02:55:14.665026 20739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:14.665169 20739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee0a5800a224e32bce95bb0c762767b, State: Initialized, Role: FOLLOWER
I20260925 02:55:14.665762 20739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee0a5800a224e32bce95bb0c762767b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43303 } }
I20260925 02:55:14.666038 20739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:14.666186 20739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:14.666361 20739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:14.668669 20739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee0a5800a224e32bce95bb0c762767b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43303 } }
I20260925 02:55:14.669170 20739 leader_election.cc:302] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ee0a5800a224e32bce95bb0c762767b; no voters:
I20260925 02:55:14.669770 20739 leader_election.cc:288] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:14.669907 20744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:14.671018 20744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [term 1 LEADER]: Becoming Leader. State: Replica: 8ee0a5800a224e32bce95bb0c762767b, State: Running, Role: LEADER
I20260925 02:55:14.671653 20744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee0a5800a224e32bce95bb0c762767b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43303 } }
I20260925 02:55:14.672101 20739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:14.674201 20746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee0a5800a224e32bce95bb0c762767b. Latest consensus state: current_term: 1 leader_uuid: "8ee0a5800a224e32bce95bb0c762767b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee0a5800a224e32bce95bb0c762767b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43303 } } }
I20260925 02:55:14.674501 20745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ee0a5800a224e32bce95bb0c762767b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee0a5800a224e32bce95bb0c762767b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43303 } } }
I20260925 02:55:14.674762 20746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:14.674791 20745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:14.676816 20751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:14.682209 20751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:14.692093 20751 catalog_manager.cc:1347] Generated new cluster ID: 5da6c15f921c4c2fb3128868e98f2ce3
I20260925 02:55:14.692231 20751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:14.713846 20751 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:14.715247 20751 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:14.725678 20751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b: Generated new TSK 0
I20260925 02:55:14.726594 20751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:14.737571 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43303
--builtin_ntp_servers=127.2.172.84:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:55:15.093287 20763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:15.093596 20763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:15.093685 20763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:15.101243 20763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:15.101351 20763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:15.101459 20763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:55:15.110754 20763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:15.112046 20763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:15.113858 20763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:15.127316 20769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:15.127496 20771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:15.127449 20768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:15.128536 20763 server_base.cc:1034] running on GCE node
I20260925 02:55:15.129186 20763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:15.130225 20763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:15.131443 20763 hybrid_clock.cc:648] HybridClock initialized: now 1790304915131389 us; error 48 us; skew 500 ppm
I20260925 02:55:15.131814 20763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:55:15.134171 20763 webserver.cc:460] Webserver started at http://127.2.172.65:42399/ using document root <none> and password file <none>
I20260925 02:55:15.134896 20763 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:15.135026 20763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:15.135363 20763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:15.138371 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "be868b4f2548474ca9c9f672abd485ec"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-g1xh"
server_key: "ad6970e43fd4da40162546ec0bff4fb8"
server_key_iv: "da58d0c67b4011660dbb22aac041db63"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.139206 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "be868b4f2548474ca9c9f672abd485ec"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-g1xh"
server_key: "ad6970e43fd4da40162546ec0bff4fb8"
server_key_iv: "da58d0c67b4011660dbb22aac041db63"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.144594 20763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:55:15.148350 20777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:15.149780 20763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:15.149983 20763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "be868b4f2548474ca9c9f672abd485ec"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-g1xh"
server_key: "ad6970e43fd4da40162546ec0bff4fb8"
server_key_iv: "da58d0c67b4011660dbb22aac041db63"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.150216 20763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:15.169633 20763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:15.170552 20763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:15.170881 20763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:15.172037 20763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:15.173733 20763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:15.173833 20763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:15.173944 20763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:15.174021 20763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:15.208734 20763 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:45075
I20260925 02:55:15.208819 20890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:45075 every 8 connection(s)
I20260925 02:55:15.210098 20763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:15.218569 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 20763
I20260925 02:55:15.218772 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:55:15.219213 2737 external_mini_cluster.cc:1428] Setting key 87435ace15fef06a3c0f6cc621d56592
I20260925 02:55:15.222437 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43303
--builtin_ntp_servers=127.2.172.84:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:55:15.223827 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.212661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:41313 (local address 127.2.172.126:43303)
0925 02:55:15.213202 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.213207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.214390 (+ 1183us) server_negotiation.cc:407] Connection header received
0925 02:55:15.216339 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.216345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.216449 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.216578 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.217669 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.218407 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.220308 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.220907 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.221089 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.222711 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.222720 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.222727 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:15.222752 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.222806 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:15.222824 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.222918 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.223198 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.223320 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":309,"threads_started":1}
I20260925 02:55:15.224749 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.213285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43303 (local address 127.2.172.65:41313)
0925 02:55:15.214211 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.214248 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:15.216053 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:15.216818 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:15.216827 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:15.216915 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:15.217485 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.217501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.218586 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.218590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.220140 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.220155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.221168 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.221174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.221465 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.222258 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:55:15.222263 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:15.222318 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:55:15.222434 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:15.222943 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:15.222953 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:15.222959 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:55:15.223145 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 02:55:15.223459 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":293,"threads_started":1}
I20260925 02:55:15.226397 20891 heartbeater.cc:347] Connected to a master server at 127.2.172.126:43303
I20260925 02:55:15.226713 20891 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:15.227471 20891 heartbeater.cc:511] Master 127.2.172.126:43303 requested a full tablet report, sending...
I20260925 02:55:15.229566 20704 ts_manager.cc:194] Registered new tserver with Master: be868b4f2548474ca9c9f672abd485ec (127.2.172.65:45075)
I20260925 02:55:15.230998 20704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.172.65:41313
W20260925 02:55:15.564920 20895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:15.565263 20895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:15.565441 20895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:15.573220 20895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:15.573354 20895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:15.573541 20895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:55:15.584784 20895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:15.586247 20895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:15.588203 20895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:15.602409 20900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:15.602388 20901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:15.602357 20903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:15.603228 20895 server_base.cc:1034] running on GCE node
I20260925 02:55:15.603938 20895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:15.604911 20895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:15.606172 20895 hybrid_clock.cc:648] HybridClock initialized: now 1790304915606120 us; error 60 us; skew 500 ppm
I20260925 02:55:15.606535 20895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:55:15.609066 20895 webserver.cc:460] Webserver started at http://127.2.172.66:43003/ using document root <none> and password file <none>
I20260925 02:55:15.609790 20895 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:15.609927 20895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:15.610255 20895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:15.613324 20895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "e84f364c30ec4718832752ef282182e9"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-g1xh"
server_key: "57e25fc6975a53d785a9c8ac228da536"
server_key_iv: "78cf06fe5e8ceeca0a014add8d51ad2a"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.614156 20895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "e84f364c30ec4718832752ef282182e9"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-g1xh"
server_key: "57e25fc6975a53d785a9c8ac228da536"
server_key_iv: "78cf06fe5e8ceeca0a014add8d51ad2a"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.619679 20895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 02:55:15.623329 20909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:15.624866 20895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:55:15.625083 20895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "e84f364c30ec4718832752ef282182e9"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-g1xh"
server_key: "57e25fc6975a53d785a9c8ac228da536"
server_key_iv: "78cf06fe5e8ceeca0a014add8d51ad2a"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.625339 20895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:15.655694 20895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:15.656585 20895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:15.656903 20895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:15.657918 20895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:15.659468 20895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:15.659629 20895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:15.659768 20895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:15.659868 20895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:15.694423 20895 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:39109
I20260925 02:55:15.694614 21022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:39109 every 8 connection(s)
I20260925 02:55:15.695914 20895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:15.701421 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 20895
I20260925 02:55:15.701633 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:55:15.702097 2737 external_mini_cluster.cc:1428] Setting key 7dc875ecbd7079fdaf83e28608a78f1c
I20260925 02:55:15.705060 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43303
--builtin_ntp_servers=127.2.172.84:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:55:15.709569 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.698618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:60431 (local address 127.2.172.126:43303)
0925 02:55:15.698873 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.698880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.700408 (+ 1528us) server_negotiation.cc:407] Connection header received
0925 02:55:15.702370 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.702377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.702481 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.702597 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.703867 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.704724 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.706398 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.706924 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.707254 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.708748 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.708753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.708756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:15.708774 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.708804 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:15.708816 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.708919 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.709158 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.709259 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 02:55:15.710644 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.699225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43303 (local address 127.2.172.66:60431)
0925 02:55:15.700235 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.700274 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:15.702054 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:15.702883 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:15.702898 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:15.703028 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:15.703682 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.703708 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.704881 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.704887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.706210 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.706223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.707332 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.707339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.707674 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.708422 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:55:15.708425 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:15.708470 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:55:15.708547 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:15.708918 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:15.708932 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:15.708942 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 02:55:15.709131 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 02:55:15.709541 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":343,"threads_started":1}
I20260925 02:55:15.712213 21023 heartbeater.cc:347] Connected to a master server at 127.2.172.126:43303
I20260925 02:55:15.712561 21023 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:15.713459 21023 heartbeater.cc:511] Master 127.2.172.126:43303 requested a full tablet report, sending...
I20260925 02:55:15.715023 20704 ts_manager.cc:194] Registered new tserver with Master: e84f364c30ec4718832752ef282182e9 (127.2.172.66:39109)
I20260925 02:55:15.715900 20704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.172.66:60431
W20260925 02:55:16.043830 21026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:16.044162 21026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:16.044323 21026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:16.051687 21026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:16.051820 21026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:16.051999 21026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:55:16.061622 21026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:16.063040 21026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:16.064816 21026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:16.078024 21031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:16.078152 21032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:16.078733 21034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:16.079195 21026 server_base.cc:1034] running on GCE node
I20260925 02:55:16.080210 21026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:16.081245 21026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:16.082522 21026 hybrid_clock.cc:648] HybridClock initialized: now 1790304916082476 us; error 59 us; skew 500 ppm
I20260925 02:55:16.082859 21026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:55:16.085202 21026 webserver.cc:460] Webserver started at http://127.2.172.67:37089/ using document root <none> and password file <none>
I20260925 02:55:16.085937 21026 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:16.086084 21026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:16.086414 21026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:16.089458 21026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1"
format_stamp: "Formatted at 2026-09-25 02:55:16 on dist-test-slave-g1xh"
server_key: "954f32f0e02593b00327802024b2308f"
server_key_iv: "42af4e731c88bbdba0218ad339a215cd"
server_key_version: "encryptionkey@0"
I20260925 02:55:16.090332 21026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1"
format_stamp: "Formatted at 2026-09-25 02:55:16 on dist-test-slave-g1xh"
server_key: "954f32f0e02593b00327802024b2308f"
server_key_iv: "42af4e731c88bbdba0218ad339a215cd"
server_key_version: "encryptionkey@0"
I20260925 02:55:16.096153 21026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 02:55:16.100046 21040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:16.101554 21026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:16.101763 21026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1"
format_stamp: "Formatted at 2026-09-25 02:55:16 on dist-test-slave-g1xh"
server_key: "954f32f0e02593b00327802024b2308f"
server_key_iv: "42af4e731c88bbdba0218ad339a215cd"
server_key_version: "encryptionkey@0"
I20260925 02:55:16.102025 21026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:16.119108 21026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:16.120028 21026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:16.120347 21026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:16.121351 21026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:16.122954 21026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:16.123090 21026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:16.123229 21026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:16.123315 21026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:16.157405 21026 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:46555
I20260925 02:55:16.157455 21153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:46555 every 8 connection(s)
I20260925 02:55:16.158700 21026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:55:16.163913 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 21026
I20260925 02:55:16.164096 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:55:16.164559 2737 external_mini_cluster.cc:1428] Setting key bf6518daca0fb99a290daa0a0e981aa5
I20260925 02:55:16.171478 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.161293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:45997 (local address 127.2.172.126:43303)
0925 02:55:16.161468 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:16.161486 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:16.162727 (+ 1241us) server_negotiation.cc:407] Connection header received
0925 02:55:16.164676 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:16.164683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:16.164773 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:16.164899 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:16.166116 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.167143 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.168815 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.169233 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.169372 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.170666 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:16.170671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:16.170674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:16.170693 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:16.170723 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:16.170734 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:16.170830 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:16.171049 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 02:55:16.171146 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:55:16.172518 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.161711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43303 (local address 127.2.172.67:45997)
0925 02:55:16.162576 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:16.162611 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:16.164401 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:16.165144 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:16.165157 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:16.165284 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:16.165949 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:16.165970 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.167309 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.167314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:16.168658 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:16.168670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.169478 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.169486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:16.169704 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.170352 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:55:16.170355 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:16.170393 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:55:16.170469 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:16.170830 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:16.170842 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:16.170851 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:55:16.171038 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 02:55:16.171473 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":278,"threads_started":1}
I20260925 02:55:16.174048 21154 heartbeater.cc:347] Connected to a master server at 127.2.172.126:43303
I20260925 02:55:16.174343 21154 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:16.175125 21154 heartbeater.cc:511] Master 127.2.172.126:43303 requested a full tablet report, sending...
I20260925 02:55:16.176491 20704 ts_manager.cc:194] Registered new tserver with Master: 0c2d89a1b7cc4c79bba6145ec6bd1ec1 (127.2.172.67:46555)
I20260925 02:55:16.177313 20704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.172.67:45997
I20260925 02:55:16.179875 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:55:16.190018 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.184159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.2.172.126:43303)
0925 02:55:16.184353 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:16.184361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:16.184685 (+ 324us) server_negotiation.cc:407] Connection header received
0925 02:55:16.184877 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:16.184882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:16.184958 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:16.185058 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:16.185552 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.186259 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.187717 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.188142 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.188280 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.189306 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:16.189314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:16.189323 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:16.189342 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:16.189378 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:16.189390 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:16.189487 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:16.189630 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 02:55:16.189728 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:55:16.193626 20704 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:55:16.200579 20704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:55:16.234587 20891 heartbeater.cc:503] Master 127.2.172.126:43303 was elected leader, sending a full tablet report...
I20260925 02:55:16.237378 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.216215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:46555 (local address 127.0.0.1:49526)
0925 02:55:16.228584 (+ 12369us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:16.228602 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:16.228712 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:16.230235 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:16.230239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:16.230263 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:16.230670 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:16.230684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.232836 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.232841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:16.234693 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:16.234712 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.236494 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.236500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:16.236672 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.236709 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:55:16.236769 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:55:16.236891 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12256,"mutex_wait_us":27,"thread_start_us":12118,"threads_started":1}
I20260925 02:55:16.237955 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.214298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.2.172.66:39109)
0925 02:55:16.228206 (+ 13908us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:16.228219 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:16.228262 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:55:16.228400 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:16.228409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:16.228707 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:16.228947 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:16.229734 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.231492 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.233703 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.235178 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.237033 (+ 1855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.237311 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:16.237392 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 02:55:16.237580 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13755,"thread_start_us":13648,"threads_started":1}
I20260925 02:55:16.236924 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.213811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39109 (local address 127.0.0.1:57022)
0925 02:55:16.222760 (+ 8949us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:16.222799 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:16.222959 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:16.228938 (+ 5979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:16.228945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:16.228972 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:16.229481 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:16.229499 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.231656 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.231661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:16.233496 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:16.233513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.235410 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.235417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:16.236168 (+ 751us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.236234 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 02:55:16.236344 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:55:16.236531 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8766,"mutex_wait_us":1,"thread_start_us":8558,"threads_started":1}
I20260925 02:55:16.238162 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.213569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45075 (local address 127.0.0.1:50158)
0925 02:55:16.214286 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:16.214324 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:16.214473 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:16.231043 (+ 16570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:16.231050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:16.231076 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:16.231456 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:16.231469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.233245 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.233252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:16.235395 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:16.235418 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.237341 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.237350 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:16.237541 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.237573 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 02:55:16.237692 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 02:55:16.237814 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":3,"thread_start_us":884,"threads_started":1}
I20260925 02:55:16.238380 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.214190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.2.172.65:45075)
0925 02:55:16.230481 (+ 16291us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:16.230491 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:16.230520 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:55:16.230635 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:16.230642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:16.230828 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:16.231031 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:16.231706 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.233104 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.235647 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.237073 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.237241 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.237483 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:16.237742 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 02:55:16.237893 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16105,"thread_start_us":17007,"threads_started":1}
I20260925 02:55:16.239387 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.216643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.2.172.67:46555)
0925 02:55:16.229440 (+ 12797us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:16.229454 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:16.229482 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:55:16.229591 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:16.229597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:16.229827 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:16.230837 (+ 1010us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:16.231092 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.232686 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.234921 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.236260 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.238493 (+ 2233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.238820 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:16.238882 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 02:55:16.239029 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12653,"thread_start_us":13313,"threads_started":1}
I20260925 02:55:16.242483 20826 tablet_service.cc:1467] Processing CreateTablet for tablet d0aa9bb718614b34842bd10519513ef7 (DEFAULT_TABLE table=test-table [id=85bf8a0e5fca466f9aad4c1e36a7613a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:16.242725 20957 tablet_service.cc:1467] Processing CreateTablet for tablet d0aa9bb718614b34842bd10519513ef7 (DEFAULT_TABLE table=test-table [id=85bf8a0e5fca466f9aad4c1e36a7613a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:16.242580 21089 tablet_service.cc:1467] Processing CreateTablet for tablet d0aa9bb718614b34842bd10519513ef7 (DEFAULT_TABLE table=test-table [id=85bf8a0e5fca466f9aad4c1e36a7613a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:16.244050 21089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0aa9bb718614b34842bd10519513ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:16.244065 20957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0aa9bb718614b34842bd10519513ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:16.244063 20826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0aa9bb718614b34842bd10519513ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:16.255234 21173 tablet_bootstrap.cc:492] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1: Bootstrap starting.
I20260925 02:55:16.258493 21173 tablet_bootstrap.cc:654] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:16.258581 21174 tablet_bootstrap.cc:492] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec: Bootstrap starting.
I20260925 02:55:16.259922 21173 log.cc:824] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:16.260309 21175 tablet_bootstrap.cc:492] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9: Bootstrap starting.
I20260925 02:55:16.262195 21174 tablet_bootstrap.cc:654] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:16.263486 21174 log.cc:824] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:16.263702 21175 tablet_bootstrap.cc:654] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:16.264564 21173 tablet_bootstrap.cc:492] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1: No bootstrap required, opened a new log
I20260925 02:55:16.264863 21173 ts_tablet_manager.cc:1397] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260925 02:55:16.264860 21175 log.cc:824] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:16.266623 21174 tablet_bootstrap.cc:492] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec: No bootstrap required, opened a new log
I20260925 02:55:16.266888 21174 ts_tablet_manager.cc:1397] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 02:55:16.267293 21175 tablet_bootstrap.cc:492] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9: No bootstrap required, opened a new log
I20260925 02:55:16.267534 21175 ts_tablet_manager.cc:1397] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260925 02:55:16.274672 21175 raft_consensus.cc:348] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be868b4f2548474ca9c9f672abd485ec" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45075 } } peers { permanent_uuid: "e84f364c30ec4718832752ef282182e9" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39109 } } peers { permanent_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 46555 } }
I20260925 02:55:16.275066 21175 raft_consensus.cc:374] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:16.275231 21175 raft_consensus.cc:729] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84f364c30ec4718832752ef282182e9, State: Initialized, Role: FOLLOWER
I20260925 02:55:16.275458 21173 raft_consensus.cc:348] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be868b4f2548474ca9c9f672abd485ec" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45075 } } peers { permanent_uuid: "e84f364c30ec4718832752ef282182e9" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39109 } } peers { permanent_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 46555 } }
I20260925 02:55:16.276074 21175 consensus_queue.cc:260] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be868b4f2548474ca9c9f672abd485ec" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45075 } } peers { permanent_uuid: "e84f364c30ec4718832752ef282182e9" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39109 } } peers { permanent_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 46555 } }
I20260925 02:55:16.275957 21173 raft_consensus.cc:374] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:16.276563 21173 raft_consensus.cc:729] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2d89a1b7cc4c79bba6145ec6bd1ec1, State: Initialized, Role: FOLLOWER
I20260925 02:55:16.277200 21173 consensus_queue.cc:260] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be868b4f2548474ca9c9f672abd485ec" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45075 } } peers { permanent_uuid: "e84f364c30ec4718832752ef282182e9" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39109 } } peers { permanent_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 46555 } }
I20260925 02:55:16.277365 21174 raft_consensus.cc:348] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be868b4f2548474ca9c9f672abd485ec" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45075 } } peers { permanent_uuid: "e84f364c30ec4718832752ef282182e9" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39109 } } peers { permanent_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 46555 } }
I20260925 02:55:16.277642 21023 heartbeater.cc:503] Master 127.2.172.126:43303 was elected leader, sending a full tablet report...
I20260925 02:55:16.277861 21174 raft_consensus.cc:374] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:16.278051 21174 raft_consensus.cc:729] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be868b4f2548474ca9c9f672abd485ec, State: Initialized, Role: FOLLOWER
I20260925 02:55:16.278422 21175 ts_tablet_manager.cc:1428] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260925 02:55:16.278959 21174 consensus_queue.cc:260] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be868b4f2548474ca9c9f672abd485ec" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45075 } } peers { permanent_uuid: "e84f364c30ec4718832752ef282182e9" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39109 } } peers { permanent_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 46555 } }
I20260925 02:55:16.282415 21174 ts_tablet_manager.cc:1428] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260925 02:55:16.284251 21154 heartbeater.cc:503] Master 127.2.172.126:43303 was elected leader, sending a full tablet report...
I20260925 02:55:16.285178 21173 ts_tablet_manager.cc:1428] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1: Time spent starting tablet: real 0.020s user 0.014s sys 0.006s
W20260925 02:55:16.410471 21155 tablet.cc:2369] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:55:16.447933 21024 tablet.cc:2369] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:55:16.462620 20892 tablet.cc:2369] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:16.700275 21181 raft_consensus.cc:482] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:16.700606 21181 raft_consensus.cc:504] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be868b4f2548474ca9c9f672abd485ec" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45075 } } peers { permanent_uuid: "e84f364c30ec4718832752ef282182e9" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39109 } } peers { permanent_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 46555 } }
I20260925 02:55:16.702221 21181 leader_election.cc:288] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e84f364c30ec4718832752ef282182e9 (127.2.172.66:39109), 0c2d89a1b7cc4c79bba6145ec6bd1ec1 (127.2.172.67:46555)
I20260925 02:55:16.709715 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.703308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:45899 (local address 127.2.172.66:39109)
0925 02:55:16.703503 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:16.703511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:16.703536 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:55:16.703861 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:16.703866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:16.703984 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:16.704084 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:16.704793 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.705551 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.707708 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.708874 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.709054 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.709131 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:16.709331 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 02:55:16.709479 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:55:16.709843 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.702635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39109 (local address 127.2.172.65:45899)
0925 02:55:16.703435 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:16.703455 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:16.703644 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:16.704110 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:16.704114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:16.704137 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:16.704626 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:16.704641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.705739 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.705745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:16.707451 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:16.707473 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.709054 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.709059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:16.709174 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.709200 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:55:16.709280 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:55:16.709376 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":31,"thread_start_us":452,"threads_started":1}
I20260925 02:55:16.710273 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.702701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:44937 (local address 127.2.172.67:46555)
0925 02:55:16.702899 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:16.702905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:16.703928 (+ 1023us) server_negotiation.cc:407] Connection header received
0925 02:55:16.704089 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:16.704094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:16.704162 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:16.704253 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:16.704955 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.706072 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.708113 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.709329 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.709501 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.709566 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:16.709913 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 02:55:16.710077 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:55:16.710949 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.702756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:46555 (local address 127.2.172.65:44937)
0925 02:55:16.703812 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:16.703830 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:16.703954 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:16.704293 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:16.704298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:16.704386 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:16.704783 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:16.704795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.706248 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.706253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:16.707931 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:16.707944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.709527 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.709534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:16.709685 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.709730 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:55:16.710450 (+ 720us) client_negotiation.cc:240] Negotiation successful
0925 02:55:16.710636 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":2,"spinlock_wait_cycles":21888,"thread_start_us":809,"threads_started":1}
I20260925 02:55:16.711098 20978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0aa9bb718614b34842bd10519513ef7" candidate_uuid: "be868b4f2548474ca9c9f672abd485ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84f364c30ec4718832752ef282182e9" is_pre_election: true
I20260925 02:55:16.711714 20978 raft_consensus.cc:2454] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be868b4f2548474ca9c9f672abd485ec in term 0.
I20260925 02:55:16.712004 21109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0aa9bb718614b34842bd10519513ef7" candidate_uuid: "be868b4f2548474ca9c9f672abd485ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1" is_pre_election: true
I20260925 02:55:16.712411 21109 raft_consensus.cc:2454] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be868b4f2548474ca9c9f672abd485ec in term 0.
I20260925 02:55:16.712388 20779 leader_election.cc:302] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be868b4f2548474ca9c9f672abd485ec, e84f364c30ec4718832752ef282182e9; no voters:
I20260925 02:55:16.712797 21181 raft_consensus.cc:2781] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:55:16.712958 21181 raft_consensus.cc:482] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:55:16.713071 21181 raft_consensus.cc:3037] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:16.715250 21181 raft_consensus.cc:504] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be868b4f2548474ca9c9f672abd485ec" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45075 } } peers { permanent_uuid: "e84f364c30ec4718832752ef282182e9" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39109 } } peers { permanent_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 46555 } }
I20260925 02:55:16.715968 21181 leader_election.cc:288] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [CANDIDATE]: Term 1 election: Requested vote from peers e84f364c30ec4718832752ef282182e9 (127.2.172.66:39109), 0c2d89a1b7cc4c79bba6145ec6bd1ec1 (127.2.172.67:46555)
I20260925 02:55:16.716485 20978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0aa9bb718614b34842bd10519513ef7" candidate_uuid: "be868b4f2548474ca9c9f672abd485ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84f364c30ec4718832752ef282182e9"
I20260925 02:55:16.716703 20978 raft_consensus.cc:3037] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:16.717711 21109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0aa9bb718614b34842bd10519513ef7" candidate_uuid: "be868b4f2548474ca9c9f672abd485ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1"
I20260925 02:55:16.717998 21109 raft_consensus.cc:3037] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:16.719053 20978 raft_consensus.cc:2454] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be868b4f2548474ca9c9f672abd485ec in term 1.
I20260925 02:55:16.719444 20779 leader_election.cc:302] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be868b4f2548474ca9c9f672abd485ec, e84f364c30ec4718832752ef282182e9; no voters:
I20260925 02:55:16.719743 21181 raft_consensus.cc:2781] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:16.720254 21181 raft_consensus.cc:686] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [term 1 LEADER]: Becoming Leader. State: Replica: be868b4f2548474ca9c9f672abd485ec, State: Running, Role: LEADER
I20260925 02:55:16.720552 21109 raft_consensus.cc:2454] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be868b4f2548474ca9c9f672abd485ec in term 1.
I20260925 02:55:16.721071 21181 consensus_queue.cc:237] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be868b4f2548474ca9c9f672abd485ec" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45075 } } peers { permanent_uuid: "e84f364c30ec4718832752ef282182e9" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39109 } } peers { permanent_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 46555 } }
I20260925 02:55:16.725801 20703 catalog_manager.cc:5495] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec reported cstate change: term changed from 0 to 1, leader changed from <none> to be868b4f2548474ca9c9f672abd485ec (127.2.172.65). New cstate: current_term: 1 leader_uuid: "be868b4f2548474ca9c9f672abd485ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be868b4f2548474ca9c9f672abd485ec" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e84f364c30ec4718832752ef282182e9" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 46555 } health_report { overall_health: UNKNOWN } } }
I20260925 02:55:16.791059 21190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.784899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.2.172.65:45075)
0925 02:55:16.785444 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:16.785451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:16.785614 (+ 163us) server_negotiation.cc:407] Connection header received
0925 02:55:16.785924 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:16.785928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:16.786014 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:16.786199 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:55:16.786694 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.787409 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.788888 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.789293 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.789433 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.789935 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:55:16.790438 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:55:16.790611 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:16.790663 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 02:55:16.790743 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":270,"threads_started":1}
I20260925 02:55:16.799772 20978 raft_consensus.cc:1261] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9 [term 1 FOLLOWER]: Refusing update from remote peer be868b4f2548474ca9c9f672abd485ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:16.799775 21109 raft_consensus.cc:1261] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1 [term 1 FOLLOWER]: Refusing update from remote peer be868b4f2548474ca9c9f672abd485ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:16.800714 21181 consensus_queue.cc:1035] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c2d89a1b7cc4c79bba6145ec6bd1ec1" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:16.801046 21184 consensus_queue.cc:1035] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e84f364c30ec4718832752ef282182e9" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:16.827904 20704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:16.828150 20704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:16.831734 20704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ee0a5800a224e32bce95bb0c762767b: Sending DeleteTablet for 3 replicas of tablet d0aa9bb718614b34842bd10519513ef7
I20260925 02:55:16.832851 20826 tablet_service.cc:1514] Processing DeleteTablet for tablet d0aa9bb718614b34842bd10519513ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:16 UTC) from {username='slave'} at 127.0.0.1:50158
I20260925 02:55:16.832908 20957 tablet_service.cc:1514] Processing DeleteTablet for tablet d0aa9bb718614b34842bd10519513ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:16 UTC) from {username='slave'} at 127.0.0.1:57022
I20260925 02:55:16.834071 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 20763
I20260925 02:55:16.834046 21089 tablet_service.cc:1514] Processing DeleteTablet for tablet d0aa9bb718614b34842bd10519513ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:16 UTC) from {username='slave'} at 127.0.0.1:49526
I20260925 02:55:16.854645 21190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.835318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50170 (local address 127.2.172.65:45075)
0925 02:55:16.835483 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:16.835490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:16.845377 (+ 9887us) server_negotiation.cc:407] Connection header received
0925 02:55:16.845820 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:16.845839 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:16.845951 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:16.846030 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:16.846751 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.848024 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.849972 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.850654 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.850753 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.852680 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:16.852693 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:16.852713 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:16.853597 (+ 884us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:16.853657 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:16.853682 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:16.853761 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:16.854053 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 02:55:16.854191 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:55:16.856472 20886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:16.858369 21206 tablet_replica.cc:330] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1: stopping tablet replica
I20260925 02:55:16.858994 21206 raft_consensus.cc:2229] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:16.859642 21206 raft_consensus.cc:2258] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:16.860754 21204 tablet_replica.cc:330] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9: stopping tablet replica
I20260925 02:55:16.861302 21204 raft_consensus.cc:2229] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:16.861901 21204 raft_consensus.cc:2258] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:16.862136 21206 ts_tablet_manager.cc:1905] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:16.863940 21204 ts_tablet_manager.cc:1905] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:16.870253 21204 ts_tablet_manager.cc:1918] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:16.870431 21204 log.cc:1196] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/d0aa9bb718614b34842bd10519513ef7
I20260925 02:55:16.870859 21204 ts_tablet_manager.cc:1939] T d0aa9bb718614b34842bd10519513ef7 P e84f364c30ec4718832752ef282182e9: Deleting consensus metadata
I20260925 02:55:16.871493 21206 ts_tablet_manager.cc:1918] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:16.871728 21206 log.cc:1196] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/d0aa9bb718614b34842bd10519513ef7
I20260925 02:55:16.872337 21206 ts_tablet_manager.cc:1939] T d0aa9bb718614b34842bd10519513ef7 P 0c2d89a1b7cc4c79bba6145ec6bd1ec1: Deleting consensus metadata
I20260925 02:55:16.872704 20689 catalog_manager.cc:4841] TS e84f364c30ec4718832752ef282182e9 (127.2.172.66:39109): tablet d0aa9bb718614b34842bd10519513ef7 (table test-table [id=85bf8a0e5fca466f9aad4c1e36a7613a]) successfully deleted
I20260925 02:55:16.874266 20690 catalog_manager.cc:4841] TS 0c2d89a1b7cc4c79bba6145ec6bd1ec1 (127.2.172.67:46555): tablet d0aa9bb718614b34842bd10519513ef7 (table test-table [id=85bf8a0e5fca466f9aad4c1e36a7613a]) successfully deleted
I20260925 02:55:16.960860 21205 tablet_replica.cc:330] T d0aa9bb718614b34842bd10519513ef7 P be868b4f2548474ca9c9f672abd485ec: stopping tablet replica
I20260925 02:55:16.961148 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 20763
W20260925 02:55:16.981004 20690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.172.65:45075 (error 108)
I20260925 02:55:16.981715 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 20895
I20260925 02:55:16.983328 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.982326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45075 (local address 127.0.0.1:50172)
0925 02:55:16.983016 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:16.983148 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":429,"threads_started":1}
W20260925 02:55:16.983664 20690 catalog_manager.cc:4584] TS be868b4f2548474ca9c9f672abd485ec (127.2.172.65:45075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0aa9bb718614b34842bd10519513ef7: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45075: connect: Connection refused (error 111)
I20260925 02:55:16.989459 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:16.982558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57030 (local address 127.2.172.66:39109)
0925 02:55:16.982754 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:16.982760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:16.982783 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:55:16.983038 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:16.983046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:16.983162 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:16.983298 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:16.983782 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.984638 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.986146 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:16.986610 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:16.986753 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:16.988046 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:16.988055 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:16.988069 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:16.988657 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:16.988702 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:16.988721 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:16.988848 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:16.988994 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 02:55:16.989099 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:55:16.990115 21018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:17.037422 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:17.036959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45075 (local address 127.0.0.1:50174)
0925 02:55:17.037189 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:17.037275 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 02:55:17.038439 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:17.038072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45075 (local address 127.0.0.1:50188)
0925 02:55:17.038218 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:17.038304 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 02:55:17.080834 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 20895
I20260925 02:55:17.099085 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:17.098574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45075 (local address 127.0.0.1:50190)
0925 02:55:17.098838 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:17.098934 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 02:55:17.100067 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 21026
I20260925 02:55:17.100419 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:17.099983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45075 (local address 127.0.0.1:50200)
0925 02:55:17.100143 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:17.100249 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 02:55:17.106462 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:17.100811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.2.172.67:46555)
0925 02:55:17.100987 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:17.100992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:17.101015 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:55:17.101095 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:17.101099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:17.101195 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:17.101286 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:17.101642 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:17.102335 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:17.103672 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:17.104092 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:17.104197 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:17.105151 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:17.105160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:17.105175 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:17.105818 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:17.105865 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:17.105885 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:17.106036 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:17.106098 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 02:55:17.106190 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:55:17.107208 21149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:17.180995 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:17.180404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45075 (local address 127.0.0.1:50208)
0925 02:55:17.180689 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:17.180803 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 02:55:17.182307 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:17.181762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45075 (local address 127.0.0.1:50210)
0925 02:55:17.182053 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:17.182140 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260925 02:55:17.196574 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 21026
I20260925 02:55:17.214697 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 20673
I20260925 02:55:17.215857 20734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:17.326172 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 20673
2026-09-25T02:55:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3226 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:17 dist-test-slave-g1xh krb5kdc[21216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:17 dist-test-slave-g1xh krb5kdc[21216](info): set up 2 sockets
Sep 25 02:55:17 dist-test-slave-g1xh krb5kdc[21216](info): commencing operation
krb5kdc: starting...
W20260925 02:55:18.477890 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:18 dist-test-slave-g1xh krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:55:18.750964 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:34515
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:34515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:19.082022 21229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:19.082407 21229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:19.082572 21229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:19.090085 21229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:19.090247 21229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:19.090371 21229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:19.090489 21229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:19.090569 21229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:19.100899 21229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:34515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:34515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:19.102360 21229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:19.103730 21229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:19.115499 21234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:19.115626 21237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:19.116113 21235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:19.116856 21229 server_base.cc:1034] running on GCE node
I20260925 02:55:19.117650 21229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:19.119091 21229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:19.120332 21229 hybrid_clock.cc:648] HybridClock initialized: now 1790304919120291 us; error 33 us; skew 500 ppm
I20260925 02:55:19.120713 21229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:19 dist-test-slave-g1xh krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:19.123911 21229 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:55:19.125381 21229 webserver.cc:460] Webserver started at http://127.2.172.126:36113/ using document root <none> and password file <none>
I20260925 02:55:19.126036 21229 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:19.126137 21229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:19.126403 21229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:19.129442 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "1085d06e0faf49c6a811cd0a4e0cb763"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-g1xh"
server_key: "3143bb49c4f2f32c5a18a06a56ebd07c"
server_key_iv: "c985d300683be89d65699c664864cfef"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.130144 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "1085d06e0faf49c6a811cd0a4e0cb763"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-g1xh"
server_key: "3143bb49c4f2f32c5a18a06a56ebd07c"
server_key_iv: "c985d300683be89d65699c664864cfef"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.135349 21229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 02:55:19.139824 21244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:19.141193 21229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:19.141388 21229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "1085d06e0faf49c6a811cd0a4e0cb763"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-g1xh"
server_key: "3143bb49c4f2f32c5a18a06a56ebd07c"
server_key_iv: "c985d300683be89d65699c664864cfef"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.141608 21229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:19.163761 21229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:19.167897 21229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:19.168216 21229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:19.187762 21229 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:34515
I20260925 02:55:19.187789 21295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:34515 every 8 connection(s)
I20260925 02:55:19.189210 21229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:19.193042 21296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:19.197397 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 21229
I20260925 02:55:19.197594 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:19.198196 2737 external_mini_cluster.cc:1428] Setting key 1b699163eed8d90670328a407cc1fa56
I20260925 02:55:19.207154 21296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763: Bootstrap starting.
I20260925 02:55:19.211081 21296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:19.212522 21296 log.cc:824] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:55:19 dist-test-slave-g1xh krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:19.216185 21296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763: No bootstrap required, opened a new log
I20260925 02:55:19.223311 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:19.200330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.2.172.126:34515)
0925 02:55:19.201139 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:19.201151 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:19.201192 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:55:19.202819 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:19.202844 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:19.203263 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:19.203703 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:19.205525 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:19.206793 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:19.208710 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:19.209385 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:19.209530 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:19.215205 (+ 5675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:19.215214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:19.215230 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:19.215266 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:19.218654 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:19.219689 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:19.219697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:19.219704 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:19.219883 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:19.220324 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:19.220327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:19.220329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:19.220894 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:19.221233 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:19.221680 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 02:55:19.222122 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":387,"threads_started":1}
I20260925 02:55:19.225834 21296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1085d06e0faf49c6a811cd0a4e0cb763" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34515 } }
I20260925 02:55:19.226259 21296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:19.226406 21296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1085d06e0faf49c6a811cd0a4e0cb763, State: Initialized, Role: FOLLOWER
I20260925 02:55:19.227022 21296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1085d06e0faf49c6a811cd0a4e0cb763" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34515 } }
I20260925 02:55:19.227267 21296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:19.227412 21296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:19.227633 21296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:19.230203 21296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1085d06e0faf49c6a811cd0a4e0cb763" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34515 } }
I20260925 02:55:19.230737 21296 leader_election.cc:302] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1085d06e0faf49c6a811cd0a4e0cb763; no voters:
I20260925 02:55:19.231329 21296 leader_election.cc:288] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:19.231490 21301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:19.232288 21301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [term 1 LEADER]: Becoming Leader. State: Replica: 1085d06e0faf49c6a811cd0a4e0cb763, State: Running, Role: LEADER
I20260925 02:55:19.232874 21301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1085d06e0faf49c6a811cd0a4e0cb763" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34515 } }
I20260925 02:55:19.233548 21296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:19.235808 21304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1085d06e0faf49c6a811cd0a4e0cb763. Latest consensus state: current_term: 1 leader_uuid: "1085d06e0faf49c6a811cd0a4e0cb763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1085d06e0faf49c6a811cd0a4e0cb763" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34515 } } }
I20260925 02:55:19.236225 21304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:19.236711 21303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1085d06e0faf49c6a811cd0a4e0cb763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1085d06e0faf49c6a811cd0a4e0cb763" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34515 } } }
I20260925 02:55:19.237160 21303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:19.237447 21311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:19.243046 21311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:19.252542 21311 catalog_manager.cc:1347] Generated new cluster ID: 384c402871c442519c765f2254035c80
I20260925 02:55:19.252673 21311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:19.265327 21311 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:19.266692 21311 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:19.278836 21311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763: Generated new TSK 0
I20260925 02:55:19.280081 21311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:55:19.362372 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34515
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:19.715364 21323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:19.715762 21323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:19.715934 21323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:19.723308 21323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:19.723471 21323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:19.723696 21323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:55:19.733875 21323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:19.735332 21323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:19.736732 21323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:19.750329 21328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:19.750444 21329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:19.750420 21331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:19.751359 21323 server_base.cc:1034] running on GCE node
I20260925 02:55:19.751991 21323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:19.752883 21323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:19.754113 21323 hybrid_clock.cc:648] HybridClock initialized: now 1790304919754070 us; error 27 us; skew 500 ppm
I20260925 02:55:19.754468 21323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:19 dist-test-slave-g1xh krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:19.765218 21323 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:55:19.766933 21323 webserver.cc:460] Webserver started at http://127.2.172.65:42617/ using document root <none> and password file <none>
I20260925 02:55:19.767726 21323 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:19.767875 21323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:19.768241 21323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:19.771193 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "6ea98f62644947dbb84af7c4065bae72"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-g1xh"
server_key: "56e44d53cf11fa65e9a46579ba4e24a0"
server_key_iv: "6c5fcf89c84d251fedbc7c5a3926b7d8"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.772037 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "6ea98f62644947dbb84af7c4065bae72"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-g1xh"
server_key: "56e44d53cf11fa65e9a46579ba4e24a0"
server_key_iv: "6c5fcf89c84d251fedbc7c5a3926b7d8"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.777261 21323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 02:55:19.780907 21338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:19.782347 21323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:19.782521 21323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "6ea98f62644947dbb84af7c4065bae72"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-g1xh"
server_key: "56e44d53cf11fa65e9a46579ba4e24a0"
server_key_iv: "6c5fcf89c84d251fedbc7c5a3926b7d8"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.782752 21323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:19.803930 21323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:19.808415 21323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:19.808725 21323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:19.809723 21323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:19.811095 21323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:19.811172 21323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:19.811349 21323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:19.811411 21323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:19.845474 21323 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:37567
I20260925 02:55:19.845517 21450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:37567 every 8 connection(s)
I20260925 02:55:19.846760 21323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:19.855830 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 21323
I20260925 02:55:19.856033 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:55:19.856451 2737 external_mini_cluster.cc:1428] Setting key 7cce6779e53bd04fc38e4f5390640e8a
Sep 25 02:55:19 dist-test-slave-g1xh krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:19.867230 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:19.849385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:42305 (local address 127.2.172.126:34515)
0925 02:55:19.849887 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:19.849895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:19.850857 (+ 962us) server_negotiation.cc:407] Connection header received
0925 02:55:19.852716 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:19.852721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:19.852795 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:19.852893 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:19.854348 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:19.855084 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:19.856856 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:19.857332 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:19.857437 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:19.861508 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:19.861514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:19.861517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:19.861547 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:19.864114 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:19.864887 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:19.864899 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:19.864903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:19.865014 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:19.865460 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:19.865465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:19.865468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:19.865732 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:19.865896 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:19.866602 (+ 706us) server_negotiation.cc:299] Negotiation successful
0925 02:55:19.866837 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
I20260925 02:55:19.868273 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:19.849933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:34515 (local address 127.2.172.65:42305)
0925 02:55:19.850677 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:19.850723 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:19.852465 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:19.853105 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:19.853113 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:19.853631 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:19.854183 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:19.854200 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:19.855223 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:19.855227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:19.856698 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:19.856711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:19.857565 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:19.857580 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:19.857776 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:19.858513 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:19.858544 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:19.861246 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:19.864268 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:19.864277 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:19.864293 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:19.864754 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:19.865125 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:19.865130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:19.865134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:19.865327 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:19.865868 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:19.865876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:19.866216 (+ 340us) client_negotiation.cc:769] Sending connection context
0925 02:55:19.866490 (+ 274us) client_negotiation.cc:240] Negotiation successful
0925 02:55:19.867043 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":298,"threads_started":1}
I20260925 02:55:19.869963 21451 heartbeater.cc:347] Connected to a master server at 127.2.172.126:34515
I20260925 02:55:19.870288 21451 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:19.871021 21451 heartbeater.cc:511] Master 127.2.172.126:34515 requested a full tablet report, sending...
I20260925 02:55:19.873347 21261 ts_manager.cc:194] Registered new tserver with Master: 6ea98f62644947dbb84af7c4065bae72 (127.2.172.65:37567)
I20260925 02:55:19.875619 21261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:42305
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:55:19.914834 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34515
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:20.248003 21457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:20.248286 21457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:20.248451 21457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:20.255787 21457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:20.255897 21457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:20.256078 21457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:55:20.265916 21457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:20.267251 21457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:20.268666 21457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:20.282420 21465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:20.282457 21463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:20.282495 21462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:20.283253 21457 server_base.cc:1034] running on GCE node
I20260925 02:55:20.284021 21457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:20.284974 21457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:20.286231 21457 hybrid_clock.cc:648] HybridClock initialized: now 1790304920286174 us; error 57 us; skew 500 ppm
I20260925 02:55:20.286557 21457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:20 dist-test-slave-g1xh krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:20.289882 21457 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:55:20.291371 21457 webserver.cc:460] Webserver started at http://127.2.172.66:36645/ using document root <none> and password file <none>
I20260925 02:55:20.292176 21457 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:20.292284 21457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:20.292546 21457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:20.295444 21457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "08a68694250540799c293bcc3e1c1717"
format_stamp: "Formatted at 2026-09-25 02:55:20 on dist-test-slave-g1xh"
server_key: "e61e6823f6339028b5a714293a517f41"
server_key_iv: "1932b29d683d29946677b2d7c4dbf5e0"
server_key_version: "encryptionkey@0"
I20260925 02:55:20.296293 21457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "08a68694250540799c293bcc3e1c1717"
format_stamp: "Formatted at 2026-09-25 02:55:20 on dist-test-slave-g1xh"
server_key: "e61e6823f6339028b5a714293a517f41"
server_key_iv: "1932b29d683d29946677b2d7c4dbf5e0"
server_key_version: "encryptionkey@0"
I20260925 02:55:20.301477 21457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:55:20.304925 21472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.306382 21457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:20.306581 21457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "08a68694250540799c293bcc3e1c1717"
format_stamp: "Formatted at 2026-09-25 02:55:20 on dist-test-slave-g1xh"
server_key: "e61e6823f6339028b5a714293a517f41"
server_key_iv: "1932b29d683d29946677b2d7c4dbf5e0"
server_key_version: "encryptionkey@0"
I20260925 02:55:20.306813 21457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:20.331851 21457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:20.336630 21457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:20.336954 21457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:20.338095 21457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:20.339502 21457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:20.339596 21457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 02:55:20.339682 21457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:20.339727 21457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.381127 21457 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:38727
I20260925 02:55:20.381179 21584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:38727 every 8 connection(s)
I20260925 02:55:20.382526 21457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:20.392172 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 21457
I20260925 02:55:20.392371 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:55:20.392817 2737 external_mini_cluster.cc:1428] Setting key cc344209dc19ba029f8d3e03107b556b
Sep 25 02:55:20 dist-test-slave-g1xh krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:20.405197 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.385216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:44953 (local address 127.2.172.126:34515)
0925 02:55:20.385704 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.385709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.386724 (+ 1015us) server_negotiation.cc:407] Connection header received
0925 02:55:20.388737 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.388744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.388853 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.388975 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:20.390440 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.391456 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.393199 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.393826 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.393976 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.397830 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:20.397839 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:20.397847 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:20.397892 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:20.401633 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.402475 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.402484 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.402491 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.402658 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.403103 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.403110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.403117 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.403529 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:20.403709 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.404543 (+ 834us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.404829 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":255,"threads_started":1}
I20260925 02:55:20.406359 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.385790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:34515 (local address 127.2.172.66:44953)
0925 02:55:20.386562 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.386612 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.388466 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.389208 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.389216 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.389720 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:20.390261 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.390277 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.391658 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.391663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.393056 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.393068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.394061 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.394070 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.394315 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.395075 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:20.395099 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:20.397587 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:20.401823 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.401830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.401845 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.402325 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.402773 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.402776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.402778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.402960 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.403767 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:20.403774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:20.404144 (+ 370us) client_negotiation.cc:769] Sending connection context
0925 02:55:20.404382 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 02:55:20.404798 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":300,"threads_started":1}
I20260925 02:55:20.407969 21585 heartbeater.cc:347] Connected to a master server at 127.2.172.126:34515
I20260925 02:55:20.408339 21585 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:20.409096 21585 heartbeater.cc:511] Master 127.2.172.126:34515 requested a full tablet report, sending...
I20260925 02:55:20.410656 21261 ts_manager.cc:194] Registered new tserver with Master: 08a68694250540799c293bcc3e1c1717 (127.2.172.66:38727)
I20260925 02:55:20.411659 21261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:44953
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:55:20.456425 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34515
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:20.794437 21591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:20.794728 21591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:20.794872 21591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:20.802302 21591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:20.802414 21591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:20.802599 21591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:55:20.812451 21591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:34515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:20.813697 21591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:20.814998 21591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:20.828593 21596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:20.828704 21597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:20.828629 21599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:20.829715 21591 server_base.cc:1034] running on GCE node
I20260925 02:55:20.830348 21591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:20.831331 21591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:20.832556 21591 hybrid_clock.cc:648] HybridClock initialized: now 1790304920832494 us; error 86 us; skew 500 ppm
I20260925 02:55:20.832892 21591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:20 dist-test-slave-g1xh krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:20.836205 21591 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:55:20.837766 21591 webserver.cc:460] Webserver started at http://127.2.172.67:38193/ using document root <none> and password file <none>
I20260925 02:55:20.838493 21591 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:20.838639 21591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:20.838968 21591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:20.841840 21591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "515003b62b4046c39f8ea73bced4a860"
format_stamp: "Formatted at 2026-09-25 02:55:20 on dist-test-slave-g1xh"
server_key: "d729ab151646979692f9d4ad5a7d0931"
server_key_iv: "460272d21d655cd5a6ed7c51cd7d7d1f"
server_key_version: "encryptionkey@0"
I20260925 02:55:20.842686 21591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "515003b62b4046c39f8ea73bced4a860"
format_stamp: "Formatted at 2026-09-25 02:55:20 on dist-test-slave-g1xh"
server_key: "d729ab151646979692f9d4ad5a7d0931"
server_key_iv: "460272d21d655cd5a6ed7c51cd7d7d1f"
server_key_version: "encryptionkey@0"
I20260925 02:55:20.848057 21591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 02:55:20.851640 21606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.853207 21591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:20.853404 21591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "515003b62b4046c39f8ea73bced4a860"
format_stamp: "Formatted at 2026-09-25 02:55:20 on dist-test-slave-g1xh"
server_key: "d729ab151646979692f9d4ad5a7d0931"
server_key_iv: "460272d21d655cd5a6ed7c51cd7d7d1f"
server_key_version: "encryptionkey@0"
I20260925 02:55:20.853609 21591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:20.872217 21591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:20.876703 21591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:20.877019 21591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:20.878067 21591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:20.878666 21451 heartbeater.cc:503] Master 127.2.172.126:34515 was elected leader, sending a full tablet report...
I20260925 02:55:20.879640 21591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:20.879735 21591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.879891 21591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:20.879963 21591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.914821 21591 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:44067
I20260925 02:55:20.914865 21718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:44067 every 8 connection(s)
I20260925 02:55:20.916230 21591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:55:20.923846 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 21591
I20260925 02:55:20.924043 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:55:20.924489 2737 external_mini_cluster.cc:1428] Setting key fd03813f3c6cbdbcb8d3fe877057231b
Sep 25 02:55:20 dist-test-slave-g1xh krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:20.936021 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.918831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:46859 (local address 127.2.172.126:34515)
0925 02:55:20.919285 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.919291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.920173 (+ 882us) server_negotiation.cc:407] Connection header received
0925 02:55:20.921964 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.921970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.922061 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.922177 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:20.923625 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.924591 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.926685 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.927197 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.927309 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.930671 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:20.930676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:20.930679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:20.930711 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:20.933181 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.933940 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.933944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.933946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.934063 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.934426 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.934429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.934431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.934689 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:20.934807 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.935527 (+ 720us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.935748 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":232,"threads_started":1}
I20260925 02:55:20.936959 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.919308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:34515 (local address 127.2.172.67:46859)
0925 02:55:20.920011 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.920049 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.921707 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.922421 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.922428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.922908 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:20.923432 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.923449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.924777 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.924781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.926548 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.926562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.927435 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.927446 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.927715 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.928344 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:20.928370 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:20.930473 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:20.933306 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.933314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.933327 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.933786 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.934156 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.934159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.934161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.934336 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.934793 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:20.934798 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:20.935190 (+ 392us) client_negotiation.cc:769] Sending connection context
0925 02:55:20.935420 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 02:55:20.935856 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":293,"threads_started":1}
I20260925 02:55:20.938481 21719 heartbeater.cc:347] Connected to a master server at 127.2.172.126:34515
I20260925 02:55:20.938872 21719 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:20.939795 21719 heartbeater.cc:511] Master 127.2.172.126:34515 requested a full tablet report, sending...
I20260925 02:55:20.941162 21261 ts_manager.cc:194] Registered new tserver with Master: 515003b62b4046c39f8ea73bced4a860 (127.2.172.67:44067)
I20260925 02:55:20.941942 21261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:46859
I20260925 02:55:20.952353 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:55:20.969630 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.956714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.2.172.126:34515)
0925 02:55:20.956923 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.956929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.957324 (+ 395us) server_negotiation.cc:407] Connection header received
0925 02:55:20.957483 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.957488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.957552 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.957661 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:20.959171 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.959987 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.961343 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.961743 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.961833 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.964469 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:20.964474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:20.964477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:20.964507 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:20.967026 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.967813 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.967817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.967819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.967929 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.968378 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.968382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.968384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.968666 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:20.968847 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.969186 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.969314 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 02:55:20.974015 21261 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:55:20.980862 21261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:55:21.004752 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.993251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.2.172.66:38727)
0925 02:55:20.993962 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.993973 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.994546 (+ 573us) server_negotiation.cc:407] Connection header received
0925 02:55:20.994872 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.994882 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.995148 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.995324 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:20.996369 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.997982 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.002031 (+ 4049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.003512 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.003700 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.003935 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.004262 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.004432 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":339,"threads_started":1}
I20260925 02:55:21.004835 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.992728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:38727 (local address 127.0.0.1:58146)
0925 02:55:20.994392 (+ 1664us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.994436 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.994627 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.995674 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.995685 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.995718 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:20.996146 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.996157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.998160 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.998165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.001859 (+ 3694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.001872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.003757 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.003777 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.003985 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.004046 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 02:55:21.004185 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 02:55:21.004381 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"mutex_wait_us":6,"thread_start_us":2423,"threads_started":1}
I20260925 02:55:21.004897 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.992698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:37567 (local address 127.0.0.1:48320)
0925 02:55:20.994392 (+ 1694us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.994436 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.994627 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.996118 (+ 1491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.996126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.996150 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:20.996617 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.996631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.998483 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.998489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.001452 (+ 2963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.001468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.003998 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.004009 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.004134 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.004195 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 02:55:21.004264 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:55:21.004404 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"mutex_wait_us":121,"thread_start_us":1237,"threads_started":1}
I20260925 02:55:21.006016 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.992960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.2.172.65:37567)
0925 02:55:20.995256 (+ 2296us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.995265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.995291 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:55:20.995388 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.995396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.995650 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.995852 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:20.996835 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.998318 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.001638 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.003754 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.005350 (+ 1596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.005588 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.005646 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.005755 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2118,"thread_start_us":1969,"threads_started":1}
I20260925 02:55:21.009675 21520 tablet_service.cc:1467] Processing CreateTablet for tablet ed073ab6234a483fb92fcb36441500ca (DEFAULT_TABLE table=test-table [id=b2f09a4f47a24bb398fc1916547aa1f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:21.009845 21386 tablet_service.cc:1467] Processing CreateTablet for tablet ed073ab6234a483fb92fcb36441500ca (DEFAULT_TABLE table=test-table [id=b2f09a4f47a24bb398fc1916547aa1f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:21.010402 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.995530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:44067 (local address 127.0.0.1:44940)
0925 02:55:21.002033 (+ 6503us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.002052 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:21.002208 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:21.003168 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:21.003173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:21.003197 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:21.003479 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.003486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.005404 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.005410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.007682 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.007701 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.009576 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.009583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.009757 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.009794 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:55:21.009901 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:55:21.010044 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6380,"thread_start_us":6187,"threads_started":1}
I20260925 02:55:21.010528 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.996081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.2.172.67:44067)
0925 02:55:21.000193 (+ 4112us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.000205 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.002340 (+ 2135us) server_negotiation.cc:407] Connection header received
0925 02:55:21.002454 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.002462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.002681 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.002899 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:21.003658 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.005244 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.007918 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.009344 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.009488 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.009710 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.010001 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.010150 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3888,"thread_start_us":3703,"threads_started":1}
I20260925 02:55:21.011224 21520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed073ab6234a483fb92fcb36441500ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:21.011287 21386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed073ab6234a483fb92fcb36441500ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:21.014856 21654 tablet_service.cc:1467] Processing CreateTablet for tablet ed073ab6234a483fb92fcb36441500ca (DEFAULT_TABLE table=test-table [id=b2f09a4f47a24bb398fc1916547aa1f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:21.016322 21654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed073ab6234a483fb92fcb36441500ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:21.023883 21738 tablet_bootstrap.cc:492] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72: Bootstrap starting.
I20260925 02:55:21.027689 21738 tablet_bootstrap.cc:654] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:21.028313 21739 tablet_bootstrap.cc:492] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717: Bootstrap starting.
I20260925 02:55:21.029206 21738 log.cc:824] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:21.031432 21741 tablet_bootstrap.cc:492] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860: Bootstrap starting.
I20260925 02:55:21.031751 21739 tablet_bootstrap.cc:654] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:21.032554 21738 tablet_bootstrap.cc:492] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72: No bootstrap required, opened a new log
I20260925 02:55:21.032887 21738 ts_tablet_manager.cc:1397] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 02:55:21.033067 21739 log.cc:824] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:21.034812 21741 tablet_bootstrap.cc:654] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:21.036109 21741 log.cc:824] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:21.036331 21739 tablet_bootstrap.cc:492] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717: No bootstrap required, opened a new log
I20260925 02:55:21.036603 21739 ts_tablet_manager.cc:1397] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260925 02:55:21.039230 21741 tablet_bootstrap.cc:492] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860: No bootstrap required, opened a new log
I20260925 02:55:21.039487 21741 ts_tablet_manager.cc:1397] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 02:55:21.041451 21738 raft_consensus.cc:348] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea98f62644947dbb84af7c4065bae72" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 37567 } } peers { permanent_uuid: "08a68694250540799c293bcc3e1c1717" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 38727 } } peers { permanent_uuid: "515003b62b4046c39f8ea73bced4a860" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44067 } }
I20260925 02:55:21.041807 21738 raft_consensus.cc:374] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:21.041901 21738 raft_consensus.cc:729] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea98f62644947dbb84af7c4065bae72, State: Initialized, Role: FOLLOWER
I20260925 02:55:21.042510 21738 consensus_queue.cc:260] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea98f62644947dbb84af7c4065bae72" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 37567 } } peers { permanent_uuid: "08a68694250540799c293bcc3e1c1717" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 38727 } } peers { permanent_uuid: "515003b62b4046c39f8ea73bced4a860" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44067 } }
I20260925 02:55:21.044263 21738 ts_tablet_manager.cc:1428] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260925 02:55:21.047327 21739 raft_consensus.cc:348] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea98f62644947dbb84af7c4065bae72" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 37567 } } peers { permanent_uuid: "08a68694250540799c293bcc3e1c1717" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 38727 } } peers { permanent_uuid: "515003b62b4046c39f8ea73bced4a860" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44067 } }
I20260925 02:55:21.047780 21739 raft_consensus.cc:374] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:21.047896 21739 raft_consensus.cc:729] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a68694250540799c293bcc3e1c1717, State: Initialized, Role: FOLLOWER
I20260925 02:55:21.048527 21739 consensus_queue.cc:260] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea98f62644947dbb84af7c4065bae72" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 37567 } } peers { permanent_uuid: "08a68694250540799c293bcc3e1c1717" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 38727 } } peers { permanent_uuid: "515003b62b4046c39f8ea73bced4a860" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44067 } }
I20260925 02:55:21.049700 21585 heartbeater.cc:503] Master 127.2.172.126:34515 was elected leader, sending a full tablet report...
I20260925 02:55:21.050393 21739 ts_tablet_manager.cc:1428] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 02:55:21.051844 21741 raft_consensus.cc:348] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea98f62644947dbb84af7c4065bae72" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 37567 } } peers { permanent_uuid: "08a68694250540799c293bcc3e1c1717" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 38727 } } peers { permanent_uuid: "515003b62b4046c39f8ea73bced4a860" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44067 } }
I20260925 02:55:21.052296 21741 raft_consensus.cc:374] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:21.052419 21741 raft_consensus.cc:729] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515003b62b4046c39f8ea73bced4a860, State: Initialized, Role: FOLLOWER
I20260925 02:55:21.053195 21741 consensus_queue.cc:260] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea98f62644947dbb84af7c4065bae72" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 37567 } } peers { permanent_uuid: "08a68694250540799c293bcc3e1c1717" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 38727 } } peers { permanent_uuid: "515003b62b4046c39f8ea73bced4a860" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44067 } }
I20260925 02:55:21.054512 21719 heartbeater.cc:503] Master 127.2.172.126:34515 was elected leader, sending a full tablet report...
I20260925 02:55:21.055083 21741 ts_tablet_manager.cc:1428] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860: Time spent starting tablet: real 0.015s user 0.010s sys 0.002s
W20260925 02:55:21.099367 21452 tablet.cc:2369] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:55:21.134717 21586 tablet.cc:2369] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:55:21.167959 21720 tablet.cc:2369] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:21.229496 21745 raft_consensus.cc:482] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:21.229827 21745 raft_consensus.cc:504] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea98f62644947dbb84af7c4065bae72" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 37567 } } peers { permanent_uuid: "08a68694250540799c293bcc3e1c1717" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 38727 } } peers { permanent_uuid: "515003b62b4046c39f8ea73bced4a860" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44067 } }
I20260925 02:55:21.231609 21745 leader_election.cc:288] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea98f62644947dbb84af7c4065bae72 (127.2.172.65:37567), 515003b62b4046c39f8ea73bced4a860 (127.2.172.67:44067)
I20260925 02:55:21.239427 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.232187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:50639 (local address 127.2.172.67:44067)
0925 02:55:21.232696 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.232705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.232922 (+ 217us) server_negotiation.cc:407] Connection header received
0925 02:55:21.233243 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.233249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.233345 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.233437 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:21.233992 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.235099 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.237266 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.238506 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.238633 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.238687 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.239137 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.239231 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260925 02:55:21.239629 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.231889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:44067 (local address 127.2.172.66:50639)
0925 02:55:21.232801 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.232828 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:21.233065 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:21.233471 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:21.233476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:21.233511 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:21.233833 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.233841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.235251 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.235257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.237092 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.237106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.238701 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.238708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.238957 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.238999 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:55:21.239095 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:55:21.239231 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":1021,"threads_started":1}
I20260925 02:55:21.240527 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.232168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:47521 (local address 127.2.172.65:37567)
0925 02:55:21.232499 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.232509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.233373 (+ 864us) server_negotiation.cc:407] Connection header received
0925 02:55:21.234107 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.234113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.234196 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.234303 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:21.234982 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.236099 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.238238 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.239436 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.239651 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.239734 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.240160 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.240286 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260925 02:55:21.240563 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.232017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:37567 (local address 127.2.172.66:47521)
0925 02:55:21.233249 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.233265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:21.233393 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:21.234319 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:21.234323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:21.234348 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:21.234784 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.234803 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.236236 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.236240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.238059 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.238075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.239865 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.239871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.240006 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.240027 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:55:21.240131 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:55:21.240252 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"thread_start_us":436,"threads_started":1}
I20260925 02:55:21.240824 21674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed073ab6234a483fb92fcb36441500ca" candidate_uuid: "08a68694250540799c293bcc3e1c1717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515003b62b4046c39f8ea73bced4a860" is_pre_election: true
I20260925 02:55:21.241405 21674 raft_consensus.cc:2454] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08a68694250540799c293bcc3e1c1717 in term 0.
I20260925 02:55:21.241513 21406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed073ab6234a483fb92fcb36441500ca" candidate_uuid: "08a68694250540799c293bcc3e1c1717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea98f62644947dbb84af7c4065bae72" is_pre_election: true
I20260925 02:55:21.242012 21406 raft_consensus.cc:2454] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08a68694250540799c293bcc3e1c1717 in term 0.
I20260925 02:55:21.242127 21474 leader_election.cc:302] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a68694250540799c293bcc3e1c1717, 515003b62b4046c39f8ea73bced4a860; no voters:
I20260925 02:55:21.242607 21745 raft_consensus.cc:2781] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:55:21.242751 21745 raft_consensus.cc:482] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:55:21.242875 21745 raft_consensus.cc:3037] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:21.245291 21745 raft_consensus.cc:504] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea98f62644947dbb84af7c4065bae72" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 37567 } } peers { permanent_uuid: "08a68694250540799c293bcc3e1c1717" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 38727 } } peers { permanent_uuid: "515003b62b4046c39f8ea73bced4a860" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44067 } }
I20260925 02:55:21.245954 21745 leader_election.cc:288] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea98f62644947dbb84af7c4065bae72 (127.2.172.65:37567), 515003b62b4046c39f8ea73bced4a860 (127.2.172.67:44067)
I20260925 02:55:21.246508 21406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed073ab6234a483fb92fcb36441500ca" candidate_uuid: "08a68694250540799c293bcc3e1c1717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea98f62644947dbb84af7c4065bae72"
I20260925 02:55:21.246555 21674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed073ab6234a483fb92fcb36441500ca" candidate_uuid: "08a68694250540799c293bcc3e1c1717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515003b62b4046c39f8ea73bced4a860"
I20260925 02:55:21.246732 21406 raft_consensus.cc:3037] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:21.246770 21674 raft_consensus.cc:3037] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:21.249061 21674 raft_consensus.cc:2454] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a68694250540799c293bcc3e1c1717 in term 1.
I20260925 02:55:21.249066 21406 raft_consensus.cc:2454] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08a68694250540799c293bcc3e1c1717 in term 1.
I20260925 02:55:21.249487 21474 leader_election.cc:302] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a68694250540799c293bcc3e1c1717, 515003b62b4046c39f8ea73bced4a860; no voters:
I20260925 02:55:21.249837 21745 raft_consensus.cc:2781] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:21.250253 21745 raft_consensus.cc:686] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 1 LEADER]: Becoming Leader. State: Replica: 08a68694250540799c293bcc3e1c1717, State: Running, Role: LEADER
I20260925 02:55:21.250789 21745 consensus_queue.cc:237] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea98f62644947dbb84af7c4065bae72" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 37567 } } peers { permanent_uuid: "08a68694250540799c293bcc3e1c1717" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 38727 } } peers { permanent_uuid: "515003b62b4046c39f8ea73bced4a860" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44067 } }
I20260925 02:55:21.255458 21261 catalog_manager.cc:5495] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08a68694250540799c293bcc3e1c1717 (127.2.172.66). New cstate: current_term: 1 leader_uuid: "08a68694250540799c293bcc3e1c1717" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea98f62644947dbb84af7c4065bae72" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 37567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a68694250540799c293bcc3e1c1717" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 38727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "515003b62b4046c39f8ea73bced4a860" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44067 } health_report { overall_health: UNKNOWN } } }
I20260925 02:55:21.303938 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.298598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.2.172.66:38727)
0925 02:55:21.298810 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.298815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.298833 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:55:21.298959 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.298962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.299070 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.299201 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:55:21.299669 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.300361 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.301714 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.302114 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.302209 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.302756 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:55:21.303243 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:55:21.303369 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.303571 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.303645 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:55:21.312398 21674 raft_consensus.cc:1261] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860 [term 1 FOLLOWER]: Refusing update from remote peer 08a68694250540799c293bcc3e1c1717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:21.312480 21406 raft_consensus.cc:1261] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72 [term 1 FOLLOWER]: Refusing update from remote peer 08a68694250540799c293bcc3e1c1717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:21.313483 21749 consensus_queue.cc:1035] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea98f62644947dbb84af7c4065bae72" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:21.313828 21745 consensus_queue.cc:1035] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "515003b62b4046c39f8ea73bced4a860" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 44067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:21.324626 21752 mvcc.cc:205] Tried to move back new op lower bound from 7333088957685166080 to 7333088957447360512. Current Snapshot: MvccSnapshot[applied={T|T < 7333088957685166080}]
I20260925 02:55:21.326676 21754 mvcc.cc:205] Tried to move back new op lower bound from 7333088957685166080 to 7333088957447360512. Current Snapshot: MvccSnapshot[applied={T|T < 7333088957685166080}]
I20260925 02:55:21.329660 21753 mvcc.cc:205] Tried to move back new op lower bound from 7333088957685166080 to 7333088957447360512. Current Snapshot: MvccSnapshot[applied={T|T < 7333088957685166080}]
I20260925 02:55:21.342168 21260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:21.342483 21260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:21.346503 21260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1085d06e0faf49c6a811cd0a4e0cb763: Sending DeleteTablet for 3 replicas of tablet ed073ab6234a483fb92fcb36441500ca
I20260925 02:55:21.347851 21386 tablet_service.cc:1514] Processing DeleteTablet for tablet ed073ab6234a483fb92fcb36441500ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:21 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:48320
I20260925 02:55:21.347884 21520 tablet_service.cc:1514] Processing DeleteTablet for tablet ed073ab6234a483fb92fcb36441500ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:21 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:58146
I20260925 02:55:21.348493 21654 tablet_service.cc:1514] Processing DeleteTablet for tablet ed073ab6234a483fb92fcb36441500ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:21 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:44940
I20260925 02:55:21.349987 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 21323
I20260925 02:55:21.360893 21763 tablet_replica.cc:330] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860: stopping tablet replica
I20260925 02:55:21.361492 21763 raft_consensus.cc:2229] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:21.362138 21763 raft_consensus.cc:2258] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:21.364732 21761 tablet_replica.cc:330] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72: stopping tablet replica
I20260925 02:55:21.365233 21761 raft_consensus.cc:2229] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:21.366021 21761 raft_consensus.cc:2258] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:21.367026 21762 tablet_replica.cc:330] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717: stopping tablet replica
I20260925 02:55:21.367451 21762 raft_consensus.cc:2229] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:55:21.368225 21762 raft_consensus.cc:2258] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:21.368963 21761 ts_tablet_manager.cc:1905] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:21.370443 21762 ts_tablet_manager.cc:1905] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:21.370734 21763 ts_tablet_manager.cc:1905] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 02:55:21 dist-test-slave-g1xh krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:55:21.376612 21761 ts_tablet_manager.cc:1918] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:21.376773 21761 log.cc:1196] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/ed073ab6234a483fb92fcb36441500ca
I20260925 02:55:21.377378 21761 ts_tablet_manager.cc:1939] T ed073ab6234a483fb92fcb36441500ca P 6ea98f62644947dbb84af7c4065bae72: Deleting consensus metadata
I20260925 02:55:21.377508 21762 ts_tablet_manager.cc:1918] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:21.377665 21762 log.cc:1196] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/ed073ab6234a483fb92fcb36441500ca
I20260925 02:55:21.377995 21763 ts_tablet_manager.cc:1918] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:21.378147 21763 log.cc:1196] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/ed073ab6234a483fb92fcb36441500ca
I20260925 02:55:21.378213 21762 ts_tablet_manager.cc:1939] T ed073ab6234a483fb92fcb36441500ca P 08a68694250540799c293bcc3e1c1717: Deleting consensus metadata
I20260925 02:55:21.378633 21763 ts_tablet_manager.cc:1939] T ed073ab6234a483fb92fcb36441500ca P 515003b62b4046c39f8ea73bced4a860: Deleting consensus metadata
I20260925 02:55:21.379323 21245 catalog_manager.cc:4841] TS 6ea98f62644947dbb84af7c4065bae72 (127.2.172.65:37567): tablet ed073ab6234a483fb92fcb36441500ca (table test-table [id=b2f09a4f47a24bb398fc1916547aa1f6]) successfully deleted
I20260925 02:55:21.380309 21246 catalog_manager.cc:4841] TS 515003b62b4046c39f8ea73bced4a860 (127.2.172.67:44067): tablet ed073ab6234a483fb92fcb36441500ca (table test-table [id=b2f09a4f47a24bb398fc1916547aa1f6]) successfully deleted
I20260925 02:55:21.380563 21246 catalog_manager.cc:4841] TS 08a68694250540799c293bcc3e1c1717 (127.2.172.66:38727): tablet ed073ab6234a483fb92fcb36441500ca (table test-table [id=b2f09a4f47a24bb398fc1916547aa1f6]) successfully deleted
I20260925 02:55:21.382339 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.351301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.2.172.65:37567)
0925 02:55:21.351532 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.351541 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.360149 (+ 8608us) server_negotiation.cc:407] Connection header received
0925 02:55:21.360412 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.360421 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.360530 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.360640 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:21.362985 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.364637 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.367258 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.367949 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.368059 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.374194 (+ 6135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:21.374215 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:21.374232 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:21.374274 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:21.378818 (+ 4544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:21.379849 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:21.379858 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:21.379862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:21.380003 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:21.380612 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:21.380617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:21.380620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:21.381017 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:21.381221 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.381741 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.381954 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:55:21.383060 21446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:21.475593 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 21323
I20260925 02:55:21.495348 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 21457
Sep 25 02:55:21 dist-test-slave-g1xh krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:55:21.514467 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.496348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.2.172.66:38727)
0925 02:55:21.496556 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.496561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.496579 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:55:21.496855 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.496860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.496947 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.497041 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:21.498800 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.499840 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.501850 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.502292 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.502405 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.507762 (+ 5357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:21.507777 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:21.507796 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:21.507835 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:21.511231 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:21.512169 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:21.512194 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:21.512199 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:21.512388 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:21.512801 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:21.512808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:21.512811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:21.513192 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:21.513575 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.513854 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.514076 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:55:21.515157 21580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:21.632946 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 21457
I20260925 02:55:21.653347 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 21591
Sep 25 02:55:21 dist-test-slave-g1xh krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:55:21.669793 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.654112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.2.172.67:44067)
0925 02:55:21.654283 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.654287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.654306 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:55:21.654442 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.654446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.654521 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.654609 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:21.656081 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.657100 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.658708 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.659123 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.659224 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.663850 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:21.663866 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:21.663883 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:21.663921 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:21.666813 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:21.667767 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:21.667785 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:21.667787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:21.667948 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:21.668340 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:21.668344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:21.668346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:21.668716 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:21.668941 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.669283 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.669507 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:55:21.670575 21714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:21.761396 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 21591
I20260925 02:55:21.779865 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 21229
I20260925 02:55:21.780892 21291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:21.893052 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 21229
2026-09-25T02:55:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4567 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T02:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:21Z Disabled control of system clock
I20260925 02:55:21.976468 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:37295
--webserver_interface=127.2.172.126
--webserver_port=0
--builtin_ntp_servers=127.2.172.84:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:37295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:55:22.308717 21772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:22.309031 21772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:22.309163 21772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:22.316718 21772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:22.316840 21772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:22.316931 21772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:22.317039 21772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:22.317117 21772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:22.327394 21772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:37295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:37295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:22.328819 21772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:22.330730 21772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:22.343492 21777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:22.343441 21780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:22.343408 21778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:22.344304 21772 server_base.cc:1034] running on GCE node
I20260925 02:55:22.345171 21772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:22.346663 21772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:22.347885 21772 hybrid_clock.cc:648] HybridClock initialized: now 1790304922347803 us; error 81 us; skew 500 ppm
I20260925 02:55:22.348269 21772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:55:22.350782 21772 webserver.cc:460] Webserver started at http://127.2.172.126:42277/ using document root <none> and password file <none>
I20260925 02:55:22.351745 21772 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:22.351873 21772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:22.352226 21772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:22.355252 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "097c1f1d3d234e00af137b191657732c"
format_stamp: "Formatted at 2026-09-25 02:55:22 on dist-test-slave-g1xh"
server_key: "ac2e7efdf9dd787bf306f76f50af385f"
server_key_iv: "798a82913a50fd0a187e7aa276c78640"
server_key_version: "encryptionkey@0"
I20260925 02:55:22.356104 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "097c1f1d3d234e00af137b191657732c"
format_stamp: "Formatted at 2026-09-25 02:55:22 on dist-test-slave-g1xh"
server_key: "ac2e7efdf9dd787bf306f76f50af385f"
server_key_iv: "798a82913a50fd0a187e7aa276c78640"
server_key_version: "encryptionkey@0"
I20260925 02:55:22.361399 21772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 02:55:22.364846 21786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:22.366396 21772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:22.366640 21772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "097c1f1d3d234e00af137b191657732c"
format_stamp: "Formatted at 2026-09-25 02:55:22 on dist-test-slave-g1xh"
server_key: "ac2e7efdf9dd787bf306f76f50af385f"
server_key_iv: "798a82913a50fd0a187e7aa276c78640"
server_key_version: "encryptionkey@0"
I20260925 02:55:22.366879 21772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:22.378921 21772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:22.380004 21772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:22.380306 21772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:22.399626 21772 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:37295
I20260925 02:55:22.399660 21837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:37295 every 8 connection(s)
I20260925 02:55:22.401078 21772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:22.401857 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 21772
I20260925 02:55:22.402050 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:22.402623 2737 external_mini_cluster.cc:1428] Setting key 860454d7d3f75251d92cdd457a851275
I20260925 02:55:22.405673 21838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:22.416730 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:22.404551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.2.172.126:37295)
0925 02:55:22.405567 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:22.405578 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:22.405620 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:55:22.407364 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:22.407387 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:22.407899 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:22.408230 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:22.408905 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:22.410193 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:22.412139 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:22.412829 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:22.412962 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:22.414275 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:22.414283 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:22.414295 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:22.414891 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:22.414964 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:22.414987 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:22.415084 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:22.415329 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 02:55:22.415719 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":378,"threads_started":1}
I20260925 02:55:22.418322 21838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c: Bootstrap starting.
I20260925 02:55:22.421797 21838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:22.422978 21838 log.cc:824] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:22.425690 21838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c: No bootstrap required, opened a new log
I20260925 02:55:22.433030 21838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097c1f1d3d234e00af137b191657732c" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 37295 } }
I20260925 02:55:22.433399 21838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:22.433516 21838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 097c1f1d3d234e00af137b191657732c, State: Initialized, Role: FOLLOWER
I20260925 02:55:22.434041 21838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097c1f1d3d234e00af137b191657732c" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 37295 } }
I20260925 02:55:22.434285 21838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:22.434429 21838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:22.434615 21838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:22.436863 21838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097c1f1d3d234e00af137b191657732c" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 37295 } }
I20260925 02:55:22.437445 21838 leader_election.cc:302] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 097c1f1d3d234e00af137b191657732c; no voters:
I20260925 02:55:22.438078 21838 leader_election.cc:288] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:22.438230 21843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:22.439133 21843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [term 1 LEADER]: Becoming Leader. State: Replica: 097c1f1d3d234e00af137b191657732c, State: Running, Role: LEADER
I20260925 02:55:22.439706 21843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097c1f1d3d234e00af137b191657732c" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 37295 } }
I20260925 02:55:22.440536 21838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:22.442543 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 097c1f1d3d234e00af137b191657732c. Latest consensus state: current_term: 1 leader_uuid: "097c1f1d3d234e00af137b191657732c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097c1f1d3d234e00af137b191657732c" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 37295 } } }
I20260925 02:55:22.442776 21844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "097c1f1d3d234e00af137b191657732c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097c1f1d3d234e00af137b191657732c" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 37295 } } }
I20260925 02:55:22.443005 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:22.443045 21844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:22.445346 21849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:22.450845 21849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:22.459529 21849 catalog_manager.cc:1347] Generated new cluster ID: 0acaac42c9224dc78d4cc972ce167687
I20260925 02:55:22.459674 21849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:22.469138 21849 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:22.470491 21849 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:22.480495 21849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c: Generated new TSK 0
I20260925 02:55:22.481355 21849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:22.490003 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:37295
--builtin_ntp_servers=127.2.172.84:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:55:22.836252 21862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:22.836576 21862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:22.836707 21862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:22.844045 21862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:22.844183 21862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:22.844288 21862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:55:22.853416 21862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:37295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:22.854660 21862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:22.856422 21862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:22.869735 21867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:22.869774 21868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:22.869807 21870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:22.870507 21862 server_base.cc:1034] running on GCE node
I20260925 02:55:22.871134 21862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:22.872193 21862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:22.873420 21862 hybrid_clock.cc:648] HybridClock initialized: now 1790304922873361 us; error 58 us; skew 500 ppm
I20260925 02:55:22.873745 21862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:55:22.876096 21862 webserver.cc:460] Webserver started at http://127.2.172.65:37443/ using document root <none> and password file <none>
I20260925 02:55:22.876791 21862 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:22.876929 21862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:22.877250 21862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:22.880187 21862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "767aeb3551aa4110952bc75b06c1f0f6"
format_stamp: "Formatted at 2026-09-25 02:55:22 on dist-test-slave-g1xh"
server_key: "a03604cc4cd960bdf5cb98f0bc43269a"
server_key_iv: "646519abeefd3adc9c624cd3630bedba"
server_key_version: "encryptionkey@0"
I20260925 02:55:22.880980 21862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "767aeb3551aa4110952bc75b06c1f0f6"
format_stamp: "Formatted at 2026-09-25 02:55:22 on dist-test-slave-g1xh"
server_key: "a03604cc4cd960bdf5cb98f0bc43269a"
server_key_iv: "646519abeefd3adc9c624cd3630bedba"
server_key_version: "encryptionkey@0"
I20260925 02:55:22.886276 21862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:55:22.889950 21876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:22.891232 21862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:55:22.891412 21862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "767aeb3551aa4110952bc75b06c1f0f6"
format_stamp: "Formatted at 2026-09-25 02:55:22 on dist-test-slave-g1xh"
server_key: "a03604cc4cd960bdf5cb98f0bc43269a"
server_key_iv: "646519abeefd3adc9c624cd3630bedba"
server_key_version: "encryptionkey@0"
I20260925 02:55:22.891702 21862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:22.936182 21862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:22.937067 21862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:22.937381 21862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:22.938479 21862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:22.940193 21862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:22.940320 21862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:22.940460 21862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:22.940559 21862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:22.974681 21862 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:41739
I20260925 02:55:22.974738 21988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:41739 every 8 connection(s)
I20260925 02:55:22.976143 21862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:22.980813 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 21862
I20260925 02:55:22.980990 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:55:22.981448 2737 external_mini_cluster.cc:1428] Setting key 8a1c2ee666f34a97dfe1b2da96690cb0
I20260925 02:55:22.984370 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:37295
--builtin_ntp_servers=127.2.172.84:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:55:22.991677 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:22.979450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:37295 (local address 127.2.172.65:46457)
0925 02:55:22.980408 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:22.980450 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:22.982383 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:22.983208 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:22.983219 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:22.983347 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:22.984014 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:22.984032 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:22.985181 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:22.985185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:22.986836 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:22.986854 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:22.988072 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:22.988080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:22.988401 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:22.989209 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:55:22.989214 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:22.989263 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:55:22.989360 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:22.989793 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:22.989805 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:22.989815 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 02:55:22.989974 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 02:55:22.990368 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":307,"threads_started":1}
I20260925 02:55:22.992244 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:22.978903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:46457 (local address 127.2.172.126:37295)
0925 02:55:22.979475 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:22.979480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:22.980725 (+ 1245us) server_negotiation.cc:407] Connection header received
0925 02:55:22.982693 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:22.982702 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:22.982802 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:22.982924 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:22.984181 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:22.985008 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:22.987020 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:22.987679 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:22.987807 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:22.989573 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:22.989581 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:22.989588 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:22.989613 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:22.989658 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:22.989674 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:22.989741 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:22.991678 (+ 1937us) server_negotiation.cc:299] Negotiation successful
0925 02:55:22.991791 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":323,"threads_started":1}
I20260925 02:55:22.993745 21989 heartbeater.cc:347] Connected to a master server at 127.2.172.126:37295
I20260925 02:55:22.994081 21989 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:22.995042 21989 heartbeater.cc:511] Master 127.2.172.126:37295 requested a full tablet report, sending...
I20260925 02:55:22.997421 21803 ts_manager.cc:194] Registered new tserver with Master: 767aeb3551aa4110952bc75b06c1f0f6 (127.2.172.65:41739)
I20260925 02:55:22.999609 21803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.172.65:46457
W20260925 02:55:23.337900 21993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:23.338186 21993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:23.338274 21993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:23.345811 21993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:23.345912 21993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:23.346020 21993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:55:23.355798 21993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:37295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:23.357071 21993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:23.358835 21993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:23.372144 21998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:23.372215 21999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:23.372421 22001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:23.373248 21993 server_base.cc:1034] running on GCE node
I20260925 02:55:23.373893 21993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:23.374913 21993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:23.376157 21993 hybrid_clock.cc:648] HybridClock initialized: now 1790304923376110 us; error 58 us; skew 500 ppm
I20260925 02:55:23.376506 21993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:55:23.378799 21993 webserver.cc:460] Webserver started at http://127.2.172.66:34749/ using document root <none> and password file <none>
I20260925 02:55:23.379520 21993 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:23.379721 21993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:23.380072 21993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:23.382948 21993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "dc38911048d04adeae749024d909e1be"
format_stamp: "Formatted at 2026-09-25 02:55:23 on dist-test-slave-g1xh"
server_key: "3f91aa085232cf1edb4ff7d3b3a25a36"
server_key_iv: "e857f042b4bb43281bcffb8a26d898b5"
server_key_version: "encryptionkey@0"
I20260925 02:55:23.383819 21993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "dc38911048d04adeae749024d909e1be"
format_stamp: "Formatted at 2026-09-25 02:55:23 on dist-test-slave-g1xh"
server_key: "3f91aa085232cf1edb4ff7d3b3a25a36"
server_key_iv: "e857f042b4bb43281bcffb8a26d898b5"
server_key_version: "encryptionkey@0"
I20260925 02:55:23.388972 21993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:55:23.392535 22007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:23.393887 21993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:55:23.394090 21993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "dc38911048d04adeae749024d909e1be"
format_stamp: "Formatted at 2026-09-25 02:55:23 on dist-test-slave-g1xh"
server_key: "3f91aa085232cf1edb4ff7d3b3a25a36"
server_key_iv: "e857f042b4bb43281bcffb8a26d898b5"
server_key_version: "encryptionkey@0"
I20260925 02:55:23.394335 21993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:23.414829 21993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:23.416010 21993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:23.416340 21993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:23.417600 21993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:23.419420 21993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:23.419536 21993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:23.419668 21993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:23.419737 21993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:23.454061 21993 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:39545
I20260925 02:55:23.454114 22119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:39545 every 8 connection(s)
I20260925 02:55:23.455471 21993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:23.465719 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 21993
I20260925 02:55:23.465912 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:55:23.466413 2737 external_mini_cluster.cc:1428] Setting key 15bb80227818e534f165ddf99988701c
I20260925 02:55:23.468281 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:23.458062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:56287 (local address 127.2.172.126:37295)
0925 02:55:23.458261 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:23.458265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:23.459701 (+ 1436us) server_negotiation.cc:407] Connection header received
0925 02:55:23.461571 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:23.461577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:23.461669 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:23.461780 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:23.462822 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.463613 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.465348 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.465845 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.465990 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:23.467422 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:23.467428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:23.467431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:23.467452 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:23.467482 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:23.467495 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:23.467622 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:23.467845 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 02:55:23.467946 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:55:23.469219 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:37295
--builtin_ntp_servers=127.2.172.84:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:55:23.469415 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:23.458596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:37295 (local address 127.2.172.66:56287)
0925 02:55:23.459494 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:23.459530 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:23.461298 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:23.462018 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:23.462026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:23.462108 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:23.462651 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:23.462667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.463781 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.463785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:23.465187 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:23.465200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.466097 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.466105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:23.466322 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:23.467081 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:55:23.467084 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:23.467128 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:55:23.467207 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:23.467621 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:23.467634 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:23.467645 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 02:55:23.467833 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 02:55:23.468228 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":294,"threads_started":1}
I20260925 02:55:23.470912 22120 heartbeater.cc:347] Connected to a master server at 127.2.172.126:37295
I20260925 02:55:23.471236 22120 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:23.472210 22120 heartbeater.cc:511] Master 127.2.172.126:37295 requested a full tablet report, sending...
I20260925 02:55:23.473716 21803 ts_manager.cc:194] Registered new tserver with Master: dc38911048d04adeae749024d909e1be (127.2.172.66:39545)
I20260925 02:55:23.474537 21803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.172.66:56287
W20260925 02:55:23.814168 22123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:23.814442 22123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:23.814531 22123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:23.821681 22123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:23.821779 22123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:23.821882 22123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:55:23.830816 22123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:37295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:23.832046 22123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:23.833716 22123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:23.846576 22128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:23.846616 22129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:23.846675 22131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:23.847497 22123 server_base.cc:1034] running on GCE node
I20260925 02:55:23.848146 22123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:23.849086 22123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:23.850317 22123 hybrid_clock.cc:648] HybridClock initialized: now 1790304923850282 us; error 47 us; skew 500 ppm
I20260925 02:55:23.850634 22123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:55:23.852917 22123 webserver.cc:460] Webserver started at http://127.2.172.67:44911/ using document root <none> and password file <none>
I20260925 02:55:23.853606 22123 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:23.853739 22123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:23.854071 22123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:23.856913 22123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "9767db467e4843da8e1fc16b72c7185e"
format_stamp: "Formatted at 2026-09-25 02:55:23 on dist-test-slave-g1xh"
server_key: "c3330f4cf3cbeea361e45d81073c926c"
server_key_iv: "f5af2d2fddb0fcc0d25517c7bdddc1f9"
server_key_version: "encryptionkey@0"
I20260925 02:55:23.857715 22123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "9767db467e4843da8e1fc16b72c7185e"
format_stamp: "Formatted at 2026-09-25 02:55:23 on dist-test-slave-g1xh"
server_key: "c3330f4cf3cbeea361e45d81073c926c"
server_key_iv: "f5af2d2fddb0fcc0d25517c7bdddc1f9"
server_key_version: "encryptionkey@0"
I20260925 02:55:23.862726 22123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 02:55:23.866245 22137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:23.867538 22123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:55:23.867780 22123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "9767db467e4843da8e1fc16b72c7185e"
format_stamp: "Formatted at 2026-09-25 02:55:23 on dist-test-slave-g1xh"
server_key: "c3330f4cf3cbeea361e45d81073c926c"
server_key_iv: "f5af2d2fddb0fcc0d25517c7bdddc1f9"
server_key_version: "encryptionkey@0"
I20260925 02:55:23.868009 22123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:23.905428 22123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:23.906296 22123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:23.906623 22123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:23.907692 22123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:23.909217 22123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:23.909343 22123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:23.909487 22123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:23.909577 22123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:23.943128 22123 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:35841
I20260925 02:55:23.943187 22249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:35841 every 8 connection(s)
I20260925 02:55:23.944427 22123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:55:23.949539 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 22123
I20260925 02:55:23.949720 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:55:23.950307 2737 external_mini_cluster.cc:1428] Setting key e9192566d9e1c4894bce77ab2d16b846
I20260925 02:55:23.957957 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:23.947095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:37019 (local address 127.2.172.126:37295)
0925 02:55:23.947267 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:23.947272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:23.948616 (+ 1344us) server_negotiation.cc:407] Connection header received
0925 02:55:23.951019 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:23.951025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:23.951122 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:23.951239 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:23.952563 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.953479 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.955043 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.955451 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.955615 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:23.956979 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:23.956984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:23.956987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:23.957007 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:23.957040 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:23.957053 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:23.957154 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:23.957495 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 02:55:23.957599 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:55:23.959097 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:23.947585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:37295 (local address 127.2.172.67:37019)
0925 02:55:23.948482 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:23.948518 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:23.950712 (+ 2194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:23.951479 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:23.951488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:23.951658 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:23.952377 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:23.952397 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.953645 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.953650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:23.954889 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:23.954901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.955709 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.955716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:23.955957 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:23.956607 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:55:23.956609 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:23.956651 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:55:23.956741 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:23.957266 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:23.957274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:23.957280 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:55:23.957472 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 02:55:23.957909 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":280,"threads_started":1}
I20260925 02:55:23.960530 22250 heartbeater.cc:347] Connected to a master server at 127.2.172.126:37295
I20260925 02:55:23.960842 22250 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:23.961627 22250 heartbeater.cc:511] Master 127.2.172.126:37295 requested a full tablet report, sending...
I20260925 02:55:23.963035 21803 ts_manager.cc:194] Registered new tserver with Master: 9767db467e4843da8e1fc16b72c7185e (127.2.172.67:35841)
I20260925 02:55:23.963994 21803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.172.67:37019
I20260925 02:55:23.965994 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:55:23.976818 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:23.970371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.2.172.126:37295)
0925 02:55:23.970639 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:23.970648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:23.971059 (+ 411us) server_negotiation.cc:407] Connection header received
0925 02:55:23.971249 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:23.971255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:23.971340 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:23.971451 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:23.972077 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.972767 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.974208 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.974658 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.974763 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:23.976054 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:23.976059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:23.976062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:23.976079 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:23.976109 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:23.976121 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:23.976230 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:23.976438 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 02:55:23.976483 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:55:23.981293 21802 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:55:23.988210 21802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:55:24.003372 21989 heartbeater.cc:503] Master 127.2.172.126:37295 was elected leader, sending a full tablet report...
I20260925 02:55:24.011533 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.000576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.2.172.67:35841)
0925 02:55:24.003092 (+ 2516us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:24.003099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:24.003130 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:55:24.003219 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:24.003225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:24.003431 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:24.003686 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:24.004564 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.006113 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.009088 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.010276 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.010420 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.010719 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:24.010981 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 02:55:24.011149 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2367,"thread_start_us":2202,"threads_started":1}
I20260925 02:55:24.011608 22262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.000561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:41739 (local address 127.0.0.1:53618)
0925 02:55:24.002026 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.002043 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:24.002177 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:24.003420 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:24.003440 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:24.003476 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:24.004014 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:24.004031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.006279 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.006284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:24.008983 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:24.008998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.010572 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.010577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:24.010799 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.010823 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:55:24.010917 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:55:24.011136 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"mutex_wait_us":158,"thread_start_us":313,"threads_started":1}
I20260925 02:55:24.011600 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.001311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.2.172.65:41739)
0925 02:55:24.002750 (+ 1439us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:24.002760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:24.002795 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:55:24.002911 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:24.002920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:24.003166 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:24.004264 (+ 1098us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:24.004509 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.006134 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.009200 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.010324 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.010692 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.010951 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:24.011020 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 02:55:24.011185 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248,"thread_start_us":1129,"threads_started":1}
I20260925 02:55:24.011633 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.000341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:35841 (local address 127.0.0.1:37210)
0925 02:55:24.001239 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.001286 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:24.001472 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:24.003691 (+ 2219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:24.003695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:24.003718 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:24.004256 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:24.004268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.006268 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.006277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:24.008883 (+ 2606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:24.008898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.010499 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.010506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:24.010734 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.010784 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 02:55:24.010920 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 02:55:24.011177 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":213,"thread_start_us":1501,"threads_started":1}
I20260925 02:55:24.015758 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.002439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39545 (local address 127.0.0.1:37566)
0925 02:55:24.003326 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.003346 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:24.003469 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:24.007353 (+ 3884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:24.007364 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:24.007396 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:24.007941 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:24.007955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.009808 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.009814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:24.013640 (+ 3826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:24.013653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.014932 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.014938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:24.015110 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.015143 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:55:24.015234 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:55:24.015365 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":58,"thread_start_us":1569,"threads_started":1}
I20260925 02:55:24.015471 21924 tablet_service.cc:1467] Processing CreateTablet for tablet 41f801f90e8444759f3bda51387dcafe (DEFAULT_TABLE table=test-table [id=68b474dfd57244528d706525b181aad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:24.015825 22185 tablet_service.cc:1467] Processing CreateTablet for tablet 41f801f90e8444759f3bda51387dcafe (DEFAULT_TABLE table=test-table [id=68b474dfd57244528d706525b181aad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:24.016739 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.002967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.2.172.66:39545)
0925 02:55:24.006728 (+ 3761us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:24.006739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:24.006767 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:55:24.006859 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:24.006868 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:24.007141 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:24.007329 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:24.008185 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.009657 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.013829 (+ 4172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.014739 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.014903 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.015129 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:24.016104 (+ 975us) server_negotiation.cc:299] Negotiation successful
0925 02:55:24.016293 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3602,"thread_start_us":3471,"threads_started":1}
I20260925 02:55:24.017151 22185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f801f90e8444759f3bda51387dcafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:24.017151 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f801f90e8444759f3bda51387dcafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:24.019435 22055 tablet_service.cc:1467] Processing CreateTablet for tablet 41f801f90e8444759f3bda51387dcafe (DEFAULT_TABLE table=test-table [id=68b474dfd57244528d706525b181aad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:24.020831 22055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f801f90e8444759f3bda51387dcafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:24.031672 22268 tablet_bootstrap.cc:492] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be: Bootstrap starting.
I20260925 02:55:24.035487 22268 tablet_bootstrap.cc:654] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:24.036586 22268 log.cc:824] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:24.037688 22269 tablet_bootstrap.cc:492] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6: Bootstrap starting.
I20260925 02:55:24.037688 22270 tablet_bootstrap.cc:492] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e: Bootstrap starting.
I20260925 02:55:24.041335 22269 tablet_bootstrap.cc:654] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:24.041919 22270 tablet_bootstrap.cc:654] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:24.042685 22269 log.cc:824] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:24.043081 22270 log.cc:824] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:24.045064 22268 tablet_bootstrap.cc:492] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be: No bootstrap required, opened a new log
I20260925 02:55:24.045363 22268 ts_tablet_manager.cc:1397] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260925 02:55:24.046484 22270 tablet_bootstrap.cc:492] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e: No bootstrap required, opened a new log
I20260925 02:55:24.046790 22270 ts_tablet_manager.cc:1397] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 02:55:24.050065 22269 tablet_bootstrap.cc:492] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6: No bootstrap required, opened a new log
I20260925 02:55:24.050324 22269 ts_tablet_manager.cc:1397] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260925 02:55:24.053153 22268 raft_consensus.cc:348] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9767db467e4843da8e1fc16b72c7185e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35841 } } peers { permanent_uuid: "767aeb3551aa4110952bc75b06c1f0f6" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 41739 } } peers { permanent_uuid: "dc38911048d04adeae749024d909e1be" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39545 } }
I20260925 02:55:24.054113 22268 raft_consensus.cc:374] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:24.054291 22268 raft_consensus.cc:729] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc38911048d04adeae749024d909e1be, State: Initialized, Role: FOLLOWER
I20260925 02:55:24.054975 22268 consensus_queue.cc:260] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9767db467e4843da8e1fc16b72c7185e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35841 } } peers { permanent_uuid: "767aeb3551aa4110952bc75b06c1f0f6" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 41739 } } peers { permanent_uuid: "dc38911048d04adeae749024d909e1be" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39545 } }
I20260925 02:55:24.056142 22120 heartbeater.cc:503] Master 127.2.172.126:37295 was elected leader, sending a full tablet report...
I20260925 02:55:24.056663 22268 ts_tablet_manager.cc:1428] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20260925 02:55:24.057128 22270 raft_consensus.cc:348] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9767db467e4843da8e1fc16b72c7185e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35841 } } peers { permanent_uuid: "767aeb3551aa4110952bc75b06c1f0f6" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 41739 } } peers { permanent_uuid: "dc38911048d04adeae749024d909e1be" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39545 } }
I20260925 02:55:24.057571 22270 raft_consensus.cc:374] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:24.057682 22270 raft_consensus.cc:729] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9767db467e4843da8e1fc16b72c7185e, State: Initialized, Role: FOLLOWER
I20260925 02:55:24.058252 22270 consensus_queue.cc:260] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9767db467e4843da8e1fc16b72c7185e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35841 } } peers { permanent_uuid: "767aeb3551aa4110952bc75b06c1f0f6" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 41739 } } peers { permanent_uuid: "dc38911048d04adeae749024d909e1be" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39545 } }
I20260925 02:55:24.059274 22250 heartbeater.cc:503] Master 127.2.172.126:37295 was elected leader, sending a full tablet report...
I20260925 02:55:24.059960 22270 ts_tablet_manager.cc:1428] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e: Time spent starting tablet: real 0.013s user 0.010s sys 0.006s
I20260925 02:55:24.061393 22269 raft_consensus.cc:348] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9767db467e4843da8e1fc16b72c7185e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35841 } } peers { permanent_uuid: "767aeb3551aa4110952bc75b06c1f0f6" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 41739 } } peers { permanent_uuid: "dc38911048d04adeae749024d909e1be" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39545 } }
I20260925 02:55:24.061820 22269 raft_consensus.cc:374] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:24.061972 22269 raft_consensus.cc:729] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767aeb3551aa4110952bc75b06c1f0f6, State: Initialized, Role: FOLLOWER
I20260925 02:55:24.062606 22269 consensus_queue.cc:260] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9767db467e4843da8e1fc16b72c7185e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35841 } } peers { permanent_uuid: "767aeb3551aa4110952bc75b06c1f0f6" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 41739 } } peers { permanent_uuid: "dc38911048d04adeae749024d909e1be" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39545 } }
I20260925 02:55:24.064352 22269 ts_tablet_manager.cc:1428] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
W20260925 02:55:24.196352 22251 tablet.cc:2369] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:55:24.207640 22121 tablet.cc:2369] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:55:24.228731 21990 tablet.cc:2369] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:24.241147 22275 raft_consensus.cc:482] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:24.241549 22275 raft_consensus.cc:504] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9767db467e4843da8e1fc16b72c7185e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35841 } } peers { permanent_uuid: "767aeb3551aa4110952bc75b06c1f0f6" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 41739 } } peers { permanent_uuid: "dc38911048d04adeae749024d909e1be" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39545 } }
I20260925 02:55:24.243471 22275 leader_election.cc:288] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 767aeb3551aa4110952bc75b06c1f0f6 (127.2.172.65:41739), dc38911048d04adeae749024d909e1be (127.2.172.66:39545)
I20260925 02:55:24.250303 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.243943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:55517 (local address 127.2.172.65:41739)
0925 02:55:24.244146 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:24.244152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:24.244172 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:55:24.244321 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:24.244333 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:24.244403 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:24.244506 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:24.245027 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.246000 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.248098 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.249462 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.249601 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.249650 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:24.250003 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 02:55:24.250107 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:55:24.250398 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.243818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:41739 (local address 127.2.172.67:55517)
0925 02:55:24.244021 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.244044 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:24.244168 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:24.244544 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:24.244549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:24.244586 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:24.244846 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:24.244860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.246097 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.246104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:24.247929 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:24.247941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.249623 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.249629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:24.249799 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.249837 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:55:24.249937 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:55:24.250085 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":20}
I20260925 02:55:24.251636 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.244239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:39545 (local address 127.2.172.67:35627)
0925 02:55:24.245101 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.245113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:24.245201 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:24.245511 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:24.245515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:24.245534 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:24.245848 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:24.245857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.247268 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.247272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:24.249319 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:24.249335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.250950 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.250959 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:24.251130 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.251153 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:55:24.251244 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:55:24.251327 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":1,"thread_start_us":504,"threads_started":1}
I20260925 02:55:24.251713 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.244542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:35627 (local address 127.2.172.66:39545)
0925 02:55:24.244763 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:24.244771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:24.245202 (+ 431us) server_negotiation.cc:407] Connection header received
0925 02:55:24.245290 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:24.245295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:24.245383 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:24.245475 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:24.246040 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.247113 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.249551 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.250783 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.250900 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.250946 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:24.251298 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 02:55:24.251434 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:55:24.251616 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f801f90e8444759f3bda51387dcafe" candidate_uuid: "9767db467e4843da8e1fc16b72c7185e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767aeb3551aa4110952bc75b06c1f0f6" is_pre_election: true
I20260925 02:55:24.252218 21944 raft_consensus.cc:2454] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9767db467e4843da8e1fc16b72c7185e in term 0.
I20260925 02:55:24.252876 22139 leader_election.cc:302] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 767aeb3551aa4110952bc75b06c1f0f6, 9767db467e4843da8e1fc16b72c7185e; no voters:
I20260925 02:55:24.252910 22075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f801f90e8444759f3bda51387dcafe" candidate_uuid: "9767db467e4843da8e1fc16b72c7185e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc38911048d04adeae749024d909e1be" is_pre_election: true
I20260925 02:55:24.253279 22275 raft_consensus.cc:2781] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:55:24.253428 22275 raft_consensus.cc:482] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:55:24.253539 22275 raft_consensus.cc:3037] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:24.253499 22075 raft_consensus.cc:2454] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9767db467e4843da8e1fc16b72c7185e in term 0.
I20260925 02:55:24.255992 22275 raft_consensus.cc:504] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9767db467e4843da8e1fc16b72c7185e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35841 } } peers { permanent_uuid: "767aeb3551aa4110952bc75b06c1f0f6" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 41739 } } peers { permanent_uuid: "dc38911048d04adeae749024d909e1be" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39545 } }
I20260925 02:55:24.256650 22275 leader_election.cc:288] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [CANDIDATE]: Term 1 election: Requested vote from peers 767aeb3551aa4110952bc75b06c1f0f6 (127.2.172.65:41739), dc38911048d04adeae749024d909e1be (127.2.172.66:39545)
I20260925 02:55:24.257172 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f801f90e8444759f3bda51387dcafe" candidate_uuid: "9767db467e4843da8e1fc16b72c7185e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767aeb3551aa4110952bc75b06c1f0f6"
I20260925 02:55:24.257202 22075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f801f90e8444759f3bda51387dcafe" candidate_uuid: "9767db467e4843da8e1fc16b72c7185e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc38911048d04adeae749024d909e1be"
I20260925 02:55:24.257396 21944 raft_consensus.cc:3037] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:24.257414 22075 raft_consensus.cc:3037] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:24.259706 21944 raft_consensus.cc:2454] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9767db467e4843da8e1fc16b72c7185e in term 1.
I20260925 02:55:24.259769 22075 raft_consensus.cc:2454] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9767db467e4843da8e1fc16b72c7185e in term 1.
I20260925 02:55:24.260200 22139 leader_election.cc:302] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9767db467e4843da8e1fc16b72c7185e, dc38911048d04adeae749024d909e1be; no voters:
I20260925 02:55:24.260521 22275 raft_consensus.cc:2781] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:24.260989 22275 raft_consensus.cc:686] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 1 LEADER]: Becoming Leader. State: Replica: 9767db467e4843da8e1fc16b72c7185e, State: Running, Role: LEADER
I20260925 02:55:24.261473 22275 consensus_queue.cc:237] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9767db467e4843da8e1fc16b72c7185e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35841 } } peers { permanent_uuid: "767aeb3551aa4110952bc75b06c1f0f6" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 41739 } } peers { permanent_uuid: "dc38911048d04adeae749024d909e1be" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39545 } }
I20260925 02:55:24.266088 21803 catalog_manager.cc:5495] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9767db467e4843da8e1fc16b72c7185e (127.2.172.67). New cstate: current_term: 1 leader_uuid: "9767db467e4843da8e1fc16b72c7185e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9767db467e4843da8e1fc16b72c7185e" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 35841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "767aeb3551aa4110952bc75b06c1f0f6" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 41739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc38911048d04adeae749024d909e1be" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39545 } health_report { overall_health: UNKNOWN } } }
I20260925 02:55:24.312085 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.306618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.2.172.67:35841)
0925 02:55:24.306873 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:24.306877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:24.306938 (+ 61us) server_negotiation.cc:407] Connection header received
0925 02:55:24.307169 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:24.307173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:24.307263 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:24.307396 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:55:24.307845 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.308583 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.310036 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.310451 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.310557 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.311054 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:55:24.311535 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:55:24.311792 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:24.311853 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 02:55:24.311891 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:55:24.320631 22075 raft_consensus.cc:1261] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be [term 1 FOLLOWER]: Refusing update from remote peer 9767db467e4843da8e1fc16b72c7185e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:24.320631 21944 raft_consensus.cc:1261] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6 [term 1 FOLLOWER]: Refusing update from remote peer 9767db467e4843da8e1fc16b72c7185e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:24.321676 22278 consensus_queue.cc:1035] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [LEADER]: Connected to new peer: Peer: permanent_uuid: "767aeb3551aa4110952bc75b06c1f0f6" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:24.322140 22275 consensus_queue.cc:1035] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc38911048d04adeae749024d909e1be" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:24.348150 21802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:24.348420 21802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:24.351986 21802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 097c1f1d3d234e00af137b191657732c: Sending DeleteTablet for 3 replicas of tablet 41f801f90e8444759f3bda51387dcafe
I20260925 02:55:24.353125 22185 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f801f90e8444759f3bda51387dcafe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:24 UTC) from {username='slave'} at 127.0.0.1:37210
I20260925 02:55:24.353844 21924 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f801f90e8444759f3bda51387dcafe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:24 UTC) from {username='slave'} at 127.0.0.1:53618
I20260925 02:55:24.354189 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 21862
I20260925 02:55:24.354328 22292 tablet_replica.cc:330] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e: stopping tablet replica
I20260925 02:55:24.354534 22055 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f801f90e8444759f3bda51387dcafe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:24 UTC) from {username='slave'} at 127.0.0.1:37566
I20260925 02:55:24.354758 22292 raft_consensus.cc:2229] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:55:24.355422 22292 raft_consensus.cc:2258] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:24.357797 22292 ts_tablet_manager.cc:1905] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:24.360575 22294 tablet_replica.cc:330] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be: stopping tablet replica
I20260925 02:55:24.361155 22293 tablet_replica.cc:330] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6: stopping tablet replica
I20260925 02:55:24.361701 22294 raft_consensus.cc:2229] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:24.362221 22293 raft_consensus.cc:2229] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:24.362808 22294 raft_consensus.cc:2258] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:24.363308 22293 raft_consensus.cc:2258] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:24.365760 22294 ts_tablet_manager.cc:1905] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:24.365892 22292 ts_tablet_manager.cc:1918] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:24.366173 22292 log.cc:1196] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/41f801f90e8444759f3bda51387dcafe
I20260925 02:55:24.366832 22292 ts_tablet_manager.cc:1939] T 41f801f90e8444759f3bda51387dcafe P 9767db467e4843da8e1fc16b72c7185e: Deleting consensus metadata
I20260925 02:55:24.367000 22293 ts_tablet_manager.cc:1905] T 41f801f90e8444759f3bda51387dcafe P 767aeb3551aa4110952bc75b06c1f0f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:24.368777 21787 catalog_manager.cc:4841] TS 9767db467e4843da8e1fc16b72c7185e (127.2.172.67:35841): tablet 41f801f90e8444759f3bda51387dcafe (table test-table [id=68b474dfd57244528d706525b181aad9]) successfully deleted
I20260925 02:55:24.373499 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.355933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.2.172.65:41739)
0925 02:55:24.356133 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:24.356140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:24.360082 (+ 3942us) server_negotiation.cc:407] Connection header received
0925 02:55:24.361214 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:24.361221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:24.361339 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:24.361412 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:24.362030 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.363182 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.369137 (+ 5955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.369912 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.370020 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.371609 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:24.371623 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:24.371648 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:24.372447 (+ 799us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:24.372505 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:24.372529 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:24.372618 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:24.372999 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 02:55:24.373109 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:55:24.373615 22294 ts_tablet_manager.cc:1918] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:24.373759 22294 log.cc:1196] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/41f801f90e8444759f3bda51387dcafe
I20260925 02:55:24.374385 22294 ts_tablet_manager.cc:1939] T 41f801f90e8444759f3bda51387dcafe P dc38911048d04adeae749024d909e1be: Deleting consensus metadata
I20260925 02:55:24.374807 21984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:24.376294 21788 catalog_manager.cc:4841] TS dc38911048d04adeae749024d909e1be (127.2.172.66:39545): tablet 41f801f90e8444759f3bda51387dcafe (table test-table [id=68b474dfd57244528d706525b181aad9]) successfully deleted
I20260925 02:55:24.482573 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 21862
W20260925 02:55:24.507318 21788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.172.65:41739 (error 108)
I20260925 02:55:24.507608 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 21993
I20260925 02:55:24.509364 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.508890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:41739 (local address 127.0.0.1:53632)
0925 02:55:24.509039 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.509173 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 02:55:24.509656 21788 catalog_manager.cc:4584] TS 767aeb3551aa4110952bc75b06c1f0f6 (127.2.172.65:41739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41f801f90e8444759f3bda51387dcafe: Network error: Client connection negotiation failed: client connection to 127.2.172.65:41739: connect: Connection refused (error 111)
I20260925 02:55:24.515956 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.508505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.2.172.66:39545)
0925 02:55:24.508715 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:24.508721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:24.508743 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:55:24.508946 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:24.508952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:24.509077 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:24.509197 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:24.509663 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.510850 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.512530 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.513174 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.513304 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.514335 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:24.514348 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:24.514367 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:24.515145 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:24.515205 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:24.515230 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:24.515362 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:24.515452 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 02:55:24.515594 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 02:55:24.516671 22115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:24.563349 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.562762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:41739 (local address 127.0.0.1:53640)
0925 02:55:24.563019 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.563154 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 02:55:24.564592 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.564164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:41739 (local address 127.0.0.1:53656)
0925 02:55:24.564345 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.564434 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 02:55:24.609035 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 21993
I20260925 02:55:24.625388 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.624870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:41739 (local address 127.0.0.1:53672)
0925 02:55:24.625128 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.625215 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260925 02:55:24.627869 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.627472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:41739 (local address 127.0.0.1:53680)
0925 02:55:24.627649 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.627736 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 02:55:24.634593 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 22123
I20260925 02:55:24.642397 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.635428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.2.172.67:35841)
0925 02:55:24.635665 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:24.635670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:24.635690 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:55:24.635869 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:24.635874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:24.635959 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:24.636082 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:24.636524 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.637479 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.638851 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:24.639312 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:24.639459 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:24.640806 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:24.640817 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:24.640836 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:24.641490 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:24.641542 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:24.641563 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:24.641708 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:24.641862 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 02:55:24.641976 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 02:55:24.643299 22245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:24.709257 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.708676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:41739 (local address 127.0.0.1:53690)
0925 02:55:24.708937 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.709047 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 02:55:24.710480 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.709992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:41739 (local address 127.0.0.1:53698)
0925 02:55:24.710220 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.710339 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260925 02:55:24.765034 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 22123
I20260925 02:55:24.785723 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 21772
I20260925 02:55:24.786834 21833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:24.896204 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.895468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:41739 (local address 127.0.0.1:53702)
0925 02:55:24.895701 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.895863 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 02:55:24.896524 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 21772
2026-09-25T02:55:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3002 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:25 dist-test-slave-g1xh krb5kdc[22303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:25 dist-test-slave-g1xh krb5kdc[22303](info): set up 2 sockets
Sep 25 02:55:25 dist-test-slave-g1xh krb5kdc[22303](info): commencing operation
krb5kdc: starting...
W20260925 02:55:26.038005 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:26 dist-test-slave-g1xh krb5kdc[22303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:55:26.319770 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:43211
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:43211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:26.659996 22316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:26.660300 22316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:26.660460 22316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:26.668478 22316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:26.668593 22316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:26.668706 22316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:26.668812 22316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:26.668879 22316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:26.679100 22316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:43211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:43211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:26.680512 22316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:26.681861 22316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:26.694164 22321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:26.694376 22322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:26.694700 22324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:26.695290 22316 server_base.cc:1034] running on GCE node
I20260925 02:55:26.696182 22316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:26.697687 22316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:26.698947 22316 hybrid_clock.cc:648] HybridClock initialized: now 1790304926698891 us; error 54 us; skew 500 ppm
I20260925 02:55:26.699306 22316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:26 dist-test-slave-g1xh krb5kdc[22303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304926, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:26.702513 22316 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:55:26.704226 22316 webserver.cc:460] Webserver started at http://127.2.172.126:35807/ using document root <none> and password file <none>
I20260925 02:55:26.704952 22316 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:26.705132 22316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:26.705456 22316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:26.708587 22316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "3ce601bf954e4a0c8a332c44f5415d99"
format_stamp: "Formatted at 2026-09-25 02:55:26 on dist-test-slave-g1xh"
server_key: "49ff03ee2e6c946205c275ac524320aa"
server_key_iv: "d401057b75bd8c81bbfc27cb4b7f48ed"
server_key_version: "encryptionkey@0"
I20260925 02:55:26.709380 22316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "3ce601bf954e4a0c8a332c44f5415d99"
format_stamp: "Formatted at 2026-09-25 02:55:26 on dist-test-slave-g1xh"
server_key: "49ff03ee2e6c946205c275ac524320aa"
server_key_iv: "d401057b75bd8c81bbfc27cb4b7f48ed"
server_key_version: "encryptionkey@0"
I20260925 02:55:26.714605 22316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 02:55:26.718287 22331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:26.719775 22316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:55:26.719962 22316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "3ce601bf954e4a0c8a332c44f5415d99"
format_stamp: "Formatted at 2026-09-25 02:55:26 on dist-test-slave-g1xh"
server_key: "49ff03ee2e6c946205c275ac524320aa"
server_key_iv: "d401057b75bd8c81bbfc27cb4b7f48ed"
server_key_version: "encryptionkey@0"
I20260925 02:55:26.720175 22316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:26.747180 22316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:26.751518 22316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:26.751857 22316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:26.771586 22316 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:43211
I20260925 02:55:26.771621 22382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:43211 every 8 connection(s)
I20260925 02:55:26.773017 22316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:26.776657 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 22316
I20260925 02:55:26.776868 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:26.776983 22383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:26.777396 2737 external_mini_cluster.cc:1428] Setting key 63d529c40446be482fe85f8678690a80
I20260925 02:55:26.791278 22383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99: Bootstrap starting.
Sep 25 02:55:26 dist-test-slave-g1xh krb5kdc[22303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:26.795200 22383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:26.796346 22383 log.cc:824] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:26.799827 22383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99: No bootstrap required, opened a new log
I20260925 02:55:26.803639 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:26.779593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.2.172.126:43211)
0925 02:55:26.780599 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:26.780612 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:26.780655 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:55:26.783010 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:26.783037 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:26.783524 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:26.783937 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:26.785819 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:26.787193 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:26.789208 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:26.789908 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:26.790042 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:26.794950 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:26.794960 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:26.794975 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:26.795014 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:26.798793 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:26.799577 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:26.799586 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:26.799594 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:26.799827 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:26.800264 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:26.800271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:26.800276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:26.800977 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:26.801289 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:26.801638 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 02:55:26.802124 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":463,"threads_started":1}
I20260925 02:55:26.810225 22383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce601bf954e4a0c8a332c44f5415d99" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43211 } }
I20260925 02:55:26.810580 22383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:26.810654 22383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce601bf954e4a0c8a332c44f5415d99, State: Initialized, Role: FOLLOWER
I20260925 02:55:26.811194 22383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce601bf954e4a0c8a332c44f5415d99" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43211 } }
I20260925 02:55:26.811408 22383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:26.811504 22383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:26.811682 22383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:26.814195 22383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce601bf954e4a0c8a332c44f5415d99" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43211 } }
I20260925 02:55:26.814649 22383 leader_election.cc:302] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ce601bf954e4a0c8a332c44f5415d99; no voters:
I20260925 02:55:26.815165 22383 leader_election.cc:288] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:26.815353 22388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:26.816169 22388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [term 1 LEADER]: Becoming Leader. State: Replica: 3ce601bf954e4a0c8a332c44f5415d99, State: Running, Role: LEADER
I20260925 02:55:26.816815 22388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce601bf954e4a0c8a332c44f5415d99" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43211 } }
I20260925 02:55:26.817476 22383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:26.819615 22390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ce601bf954e4a0c8a332c44f5415d99. Latest consensus state: current_term: 1 leader_uuid: "3ce601bf954e4a0c8a332c44f5415d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce601bf954e4a0c8a332c44f5415d99" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43211 } } }
I20260925 02:55:26.820088 22390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:26.819985 22389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ce601bf954e4a0c8a332c44f5415d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce601bf954e4a0c8a332c44f5415d99" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43211 } } }
I20260925 02:55:26.820386 22389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:26.823318 22395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:26.829277 22395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:26.838881 22395 catalog_manager.cc:1347] Generated new cluster ID: eed3e17aed5542078a0ae122db229b42
I20260925 02:55:26.839028 22395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:26.868785 22395 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:26.870821 22395 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:26.878595 22395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99: Generated new TSK 0
I20260925 02:55:26.879436 22395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
Entry for principal kudu/127.2.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab.
I20260925 02:55:26.950914 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43211
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:27.304889 22409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:27.305197 22409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:27.305352 22409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:27.312997 22409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:27.313122 22409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:27.313333 22409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:55:27.325042 22409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.65.keytab
--principal=kudu/127.2.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:27.326439 22409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:27.327867 22409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:27.342300 22414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:27.342425 22417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:27.342857 22415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:27.343214 22409 server_base.cc:1034] running on GCE node
I20260925 02:55:27.344127 22409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:27.345322 22409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:27.346572 22409 hybrid_clock.cc:648] HybridClock initialized: now 1790304927346508 us; error 54 us; skew 500 ppm
I20260925 02:55:27.346908 22409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:27 dist-test-slave-g1xh krb5kdc[22303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304927, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:27.350122 22409 init.cc:377] Logged in from keytab as kudu/127.2.172.65@KRBTEST.COM (short username kudu)
I20260925 02:55:27.351711 22409 webserver.cc:460] Webserver started at http://127.2.172.65:34969/ using document root <none> and password file <none>
I20260925 02:55:27.352403 22409 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:27.352556 22409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:27.352877 22409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:27.355906 22409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "14aa133b7034489f94a31c1ab5dd5a51"
format_stamp: "Formatted at 2026-09-25 02:55:27 on dist-test-slave-g1xh"
server_key: "d88507481cb3ffe354fdaf072cf0fdf7"
server_key_iv: "2090ef8fe29835bb886fddec545be1e6"
server_key_version: "encryptionkey@0"
I20260925 02:55:27.356798 22409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "14aa133b7034489f94a31c1ab5dd5a51"
format_stamp: "Formatted at 2026-09-25 02:55:27 on dist-test-slave-g1xh"
server_key: "d88507481cb3ffe354fdaf072cf0fdf7"
server_key_iv: "2090ef8fe29835bb886fddec545be1e6"
server_key_version: "encryptionkey@0"
I20260925 02:55:27.362142 22409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:55:27.365633 22424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:27.366902 22409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 02:55:27.367137 22409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "14aa133b7034489f94a31c1ab5dd5a51"
format_stamp: "Formatted at 2026-09-25 02:55:27 on dist-test-slave-g1xh"
server_key: "d88507481cb3ffe354fdaf072cf0fdf7"
server_key_iv: "2090ef8fe29835bb886fddec545be1e6"
server_key_version: "encryptionkey@0"
I20260925 02:55:27.367373 22409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:27.397804 22409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:27.401950 22409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:27.402261 22409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:27.403268 22409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:27.404762 22409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:27.404876 22409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:27.405026 22409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:27.405117 22409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:27.439780 22409 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:35911
I20260925 02:55:27.439886 22536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:35911 every 8 connection(s)
I20260925 02:55:27.441135 22409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:27.442596 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 22409
I20260925 02:55:27.442894 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:55:27.443452 2737 external_mini_cluster.cc:1428] Setting key f2af2d623699d5c97ed7852d06dad7dd
Sep 25 02:55:27 dist-test-slave-g1xh krb5kdc[22303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304927, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.65@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:27.462725 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:27.444221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:36543 (local address 127.2.172.126:43211)
0925 02:55:27.445763 (+ 1542us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:27.445767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:27.446148 (+ 381us) server_negotiation.cc:407] Connection header received
0925 02:55:27.447965 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:27.447974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:27.448063 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:27.448182 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:27.449595 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:27.450728 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:27.452341 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:27.453037 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:27.453191 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:27.456923 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:27.456928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:27.456932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:27.456964 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:27.459615 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:27.460387 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:27.460393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:27.460403 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:27.460523 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:27.460963 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:27.460973 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:27.460975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:27.461230 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:27.461385 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:27.462108 (+ 723us) server_negotiation.cc:299] Negotiation successful
0925 02:55:27.462352 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1438,"thread_start_us":1316,"threads_started":1}
I20260925 02:55:27.463776 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:27.444723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43211 (local address 127.2.172.65:36543)
0925 02:55:27.445996 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:27.446038 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:27.447715 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:27.448373 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:27.448381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:27.448870 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:27.449416 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:27.449443 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:27.450895 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:27.450899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:27.452195 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:27.452207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:27.453258 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:27.453263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:27.453469 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:27.454156 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:27.454177 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:27.456680 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:27.459787 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:27.459796 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:27.459815 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:27.460254 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:27.460636 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:27.460640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:27.460644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:27.460840 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:27.461362 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:27.461370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:27.461725 (+ 355us) client_negotiation.cc:769] Sending connection context
0925 02:55:27.461986 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 02:55:27.462541 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":370,"threads_started":1}
I20260925 02:55:27.465662 22537 heartbeater.cc:347] Connected to a master server at 127.2.172.126:43211
I20260925 02:55:27.466022 22537 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:27.466764 22537 heartbeater.cc:511] Master 127.2.172.126:43211 requested a full tablet report, sending...
I20260925 02:55:27.468942 22348 ts_manager.cc:194] Registered new tserver with Master: 14aa133b7034489f94a31c1ab5dd5a51 (127.2.172.65:35911)
I20260925 02:55:27.470427 22348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.65@KRBTEST.COM'} at 127.2.172.65:36543
WARNING: no policy specified for kudu/127.2.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
Entry for principal kudu/127.2.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab.
I20260925 02:55:27.507233 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43211
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:27.851800 22543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:27.852078 22543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:27.852172 22543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:27.859395 22543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:27.859516 22543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:27.859710 22543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:55:27.869899 22543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.66.keytab
--principal=kudu/127.2.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:27.871136 22543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:27.872483 22543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:27.886065 22549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:27.886083 22551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:27.886082 22548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:27.886559 22543 server_base.cc:1034] running on GCE node
I20260925 02:55:27.887360 22543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:27.888355 22543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:27.889567 22543 hybrid_clock.cc:648] HybridClock initialized: now 1790304927889527 us; error 28 us; skew 500 ppm
I20260925 02:55:27.889853 22543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:27 dist-test-slave-g1xh krb5kdc[22303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304927, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:27.893062 22543 init.cc:377] Logged in from keytab as kudu/127.2.172.66@KRBTEST.COM (short username kudu)
I20260925 02:55:27.894565 22543 webserver.cc:460] Webserver started at http://127.2.172.66:45217/ using document root <none> and password file <none>
I20260925 02:55:27.895273 22543 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:27.895409 22543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:27.895805 22543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:27.898747 22543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "f99871b354444b4b8dbb18e59c1cb6ee"
format_stamp: "Formatted at 2026-09-25 02:55:27 on dist-test-slave-g1xh"
server_key: "0381f8edc42dcca21b7caf22765e2121"
server_key_iv: "db358dcff98a60c8d8069263016931b1"
server_key_version: "encryptionkey@0"
I20260925 02:55:27.899586 22543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "f99871b354444b4b8dbb18e59c1cb6ee"
format_stamp: "Formatted at 2026-09-25 02:55:27 on dist-test-slave-g1xh"
server_key: "0381f8edc42dcca21b7caf22765e2121"
server_key_iv: "db358dcff98a60c8d8069263016931b1"
server_key_version: "encryptionkey@0"
I20260925 02:55:27.904786 22543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:55:27.908447 22558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:27.909803 22543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:55:27.909988 22543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "f99871b354444b4b8dbb18e59c1cb6ee"
format_stamp: "Formatted at 2026-09-25 02:55:27 on dist-test-slave-g1xh"
server_key: "0381f8edc42dcca21b7caf22765e2121"
server_key_iv: "db358dcff98a60c8d8069263016931b1"
server_key_version: "encryptionkey@0"
I20260925 02:55:27.910174 22543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:27.951159 22543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:27.955233 22543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:27.955567 22543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:27.956610 22543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:27.958016 22543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:27.958088 22543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:27.958261 22543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:27.958313 22543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:27.991495 22543 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:46399
I20260925 02:55:27.991616 22670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:46399 every 8 connection(s)
I20260925 02:55:27.992813 22543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:27.995062 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 22543
I20260925 02:55:27.995261 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:55:27.995877 2737 external_mini_cluster.cc:1428] Setting key 29abd2c7ee07e688315685085c740b0b
Sep 25 02:55:28 dist-test-slave-g1xh krb5kdc[22303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304927, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.66@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:28.018138 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:27.995323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:40971 (local address 127.2.172.126:43211)
0925 02:55:27.995847 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:27.995854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:27.996865 (+ 1011us) server_negotiation.cc:407] Connection header received
0925 02:55:27.999285 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:27.999289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:27.999353 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:27.999444 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:28.001643 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.002375 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.004994 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.005459 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.005569 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.010296 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:28.010303 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:28.010308 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:28.010359 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:28.014144 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:28.015183 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:28.015191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:28.015197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:28.015371 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:28.015935 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:28.015943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:28.015949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:28.016364 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:28.016540 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:28.017309 (+ 769us) server_negotiation.cc:299] Negotiation successful
0925 02:55:28.017616 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":265,"threads_started":1}
I20260925 02:55:28.019050 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:27.995888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43211 (local address 127.2.172.66:40971)
0925 02:55:27.996677 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:27.996716 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:27.998989 (+ 2273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:27.999764 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:27.999779 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:28.000605 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:28.001437 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.001466 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.002533 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.002541 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.004819 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.004839 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.005716 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.005726 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.006055 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.006823 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:28.006859 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:28.009963 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:28.014382 (+ 4419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:28.014391 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:28.014417 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:28.015026 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:28.015499 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:28.015502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:28.015504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:28.015805 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:28.016511 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:28.016518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:28.016912 (+ 394us) client_negotiation.cc:769] Sending connection context
0925 02:55:28.017148 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 02:55:28.017615 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":302,"threads_started":1}
I20260925 02:55:28.020804 22671 heartbeater.cc:347] Connected to a master server at 127.2.172.126:43211
I20260925 02:55:28.021128 22671 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:28.021886 22671 heartbeater.cc:511] Master 127.2.172.126:43211 requested a full tablet report, sending...
I20260925 02:55:28.023422 22348 ts_manager.cc:194] Registered new tserver with Master: f99871b354444b4b8dbb18e59c1cb6ee (127.2.172.66:46399)
I20260925 02:55:28.024420 22348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.66@KRBTEST.COM'} at 127.2.172.66:40971
WARNING: no policy specified for kudu/127.2.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
Entry for principal kudu/127.2.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab.
I20260925 02:55:28.063457 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43211
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:28.398248 22677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:28.398524 22677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:28.398746 22677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:28.406054 22677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:28.406175 22677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:28.406378 22677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:55:28.416718 22677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.67.keytab
--principal=kudu/127.2.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:43211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:28.418104 22677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:28.419505 22677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:28.433022 22683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:28.433094 22685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:28.433171 22682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:28.433720 22677 server_base.cc:1034] running on GCE node
I20260925 02:55:28.434369 22677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:28.435412 22677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:28.436640 22677 hybrid_clock.cc:648] HybridClock initialized: now 1790304928436595 us; error 60 us; skew 500 ppm
I20260925 02:55:28.437036 22677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:28 dist-test-slave-g1xh krb5kdc[22303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:28.440289 22677 init.cc:377] Logged in from keytab as kudu/127.2.172.67@KRBTEST.COM (short username kudu)
I20260925 02:55:28.441867 22677 webserver.cc:460] Webserver started at http://127.2.172.67:35265/ using document root <none> and password file <none>
I20260925 02:55:28.442554 22677 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:28.442683 22677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:28.443022 22677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:28.446010 22677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "a4815d8ac9e14606972b92d3351fb7d6"
format_stamp: "Formatted at 2026-09-25 02:55:28 on dist-test-slave-g1xh"
server_key: "3146c5f45464767f36f11cd0a3e2df75"
server_key_iv: "b2b8bed18dbd1dae335a41e9089cc48b"
server_key_version: "encryptionkey@0"
I20260925 02:55:28.446844 22677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "a4815d8ac9e14606972b92d3351fb7d6"
format_stamp: "Formatted at 2026-09-25 02:55:28 on dist-test-slave-g1xh"
server_key: "3146c5f45464767f36f11cd0a3e2df75"
server_key_iv: "b2b8bed18dbd1dae335a41e9089cc48b"
server_key_version: "encryptionkey@0"
I20260925 02:55:28.452384 22677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:55:28.455997 22692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:28.457410 22677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 02:55:28.457592 22677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "a4815d8ac9e14606972b92d3351fb7d6"
format_stamp: "Formatted at 2026-09-25 02:55:28 on dist-test-slave-g1xh"
server_key: "3146c5f45464767f36f11cd0a3e2df75"
server_key_iv: "b2b8bed18dbd1dae335a41e9089cc48b"
server_key_version: "encryptionkey@0"
I20260925 02:55:28.457793 22677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:28.470453 22677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:28.473707 22537 heartbeater.cc:503] Master 127.2.172.126:43211 was elected leader, sending a full tablet report...
I20260925 02:55:28.474645 22677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:28.474965 22677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:28.476022 22677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:28.477407 22677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:28.477516 22677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:28.477679 22677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:28.477754 22677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:28.512446 22677 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:42213
I20260925 02:55:28.512492 22804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:42213 every 8 connection(s)
I20260925 02:55:28.513859 22677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:55:28.521154 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 22677
I20260925 02:55:28.521368 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:55:28.521792 2737 external_mini_cluster.cc:1428] Setting key 1b6cefde7e4e5c551cdb36fa89c8f55f
Sep 25 02:55:28 dist-test-slave-g1xh krb5kdc[22303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.67@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:28.534577 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.516641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:51101 (local address 127.2.172.126:43211)
0925 02:55:28.516953 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:28.516958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:28.518030 (+ 1072us) server_negotiation.cc:407] Connection header received
0925 02:55:28.520085 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:28.520091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:28.520201 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:28.520315 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:28.521899 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.522795 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.524506 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.524980 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.525155 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.528600 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:28.528609 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:28.528619 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:28.528653 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:28.531255 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:28.532096 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:28.532102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:28.532105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:28.532245 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:28.532691 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:28.532696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:28.532699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:28.532989 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:28.533160 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:28.533969 (+ 809us) server_negotiation.cc:299] Negotiation successful
0925 02:55:28.534220 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260925 02:55:28.535778 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.517058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:43211 (local address 127.2.172.67:51101)
0925 02:55:28.517826 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:28.517882 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:28.519800 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:28.520556 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:28.520564 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:28.521080 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:28.521727 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.521745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.522966 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.522971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.524348 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.524361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.525261 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.525268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.525508 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.526120 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:28.526143 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:28.528382 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:28.531417 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:28.531428 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:28.531447 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:28.531964 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:28.532363 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:28.532368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:28.532372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:28.532563 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:28.533130 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:28.533138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:28.533493 (+ 355us) client_negotiation.cc:769] Sending connection context
0925 02:55:28.533771 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 02:55:28.534334 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":336,"threads_started":1}
I20260925 02:55:28.537286 22805 heartbeater.cc:347] Connected to a master server at 127.2.172.126:43211
I20260925 02:55:28.537612 22805 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:28.538317 22805 heartbeater.cc:511] Master 127.2.172.126:43211 requested a full tablet report, sending...
I20260925 02:55:28.539776 22348 ts_manager.cc:194] Registered new tserver with Master: a4815d8ac9e14606972b92d3351fb7d6 (127.2.172.67:42213)
I20260925 02:55:28.540596 22348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.172.67@KRBTEST.COM'} at 127.2.172.67:51101
I20260925 02:55:28.549770 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:55:28.566597 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.553888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.2.172.126:43211)
0925 02:55:28.554128 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:28.554135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:28.554409 (+ 274us) server_negotiation.cc:407] Connection header received
0925 02:55:28.554541 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:28.554545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:28.554617 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:28.554697 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:28.556187 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.556925 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.558184 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.558588 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.558708 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.561402 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:28.561415 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:28.561426 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:28.561461 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:28.564120 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:28.564770 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:28.564774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:28.564776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:28.564894 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:28.565243 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:28.565246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:28.565248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:28.565562 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:28.565752 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:28.566094 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 02:55:28.566251 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:55:28.570559 22348 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:55:28.577644 22348 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:55:28.609046 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.590324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.2.172.66:46399)
0925 02:55:28.598493 (+ 8169us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:28.598507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:28.598546 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:55:28.598695 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:28.598706 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:28.599014 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:28.600246 (+ 1232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:28.600554 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.602706 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.605947 (+ 3241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.607599 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.607740 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.607980 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:28.608517 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 02:55:28.608705 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7976,"thread_start_us":7825,"threads_started":1}
I20260925 02:55:28.609088 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.590135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:46399 (local address 127.0.0.1:32890)
0925 02:55:28.591154 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:28.591234 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:28.591497 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:28.599296 (+ 7799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:28.599307 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:28.599349 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:28.599948 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.599967 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.602922 (+ 2955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.602931 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.605737 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.605756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.607845 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.607854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.608136 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.608205 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 02:55:28.608402 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 02:55:28.608628 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":8412,"threads_started":1}
I20260925 02:55:28.612517 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.590838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.2.172.67:42213)
0925 02:55:28.604292 (+ 13454us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:28.604302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:28.604342 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:55:28.604452 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:28.604463 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:28.604740 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:28.604962 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:28.605790 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.607411 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.610247 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.611359 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.611519 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.611775 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:28.612031 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 02:55:28.612184 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13279,"thread_start_us":13120,"threads_started":1}
I20260925 02:55:28.612583 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.590239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:42213 (local address 127.0.0.1:48942)
0925 02:55:28.598848 (+ 8609us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:28.598870 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:28.599015 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:28.604987 (+ 5972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:28.604993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:28.605023 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:28.605544 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.605562 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.607623 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.607628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.610059 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.610072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.611621 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.611627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.611813 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.611855 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:55:28.611978 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 02:55:28.612120 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8475,"mutex_wait_us":117,"thread_start_us":414,"threads_started":1}
I20260925 02:55:28.613703 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.592410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.2.172.65:35911)
0925 02:55:28.604676 (+ 12266us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:28.604685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:28.604712 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:55:28.604810 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:28.604817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:28.605042 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:28.605241 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:28.606068 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.607410 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.610495 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.611960 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.612929 (+ 969us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.613213 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:28.613272 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 02:55:28.613389 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12078,"thread_start_us":11923,"threads_started":1}
I20260925 02:55:28.613612 22606 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3979dfe7e444a6a725be210205cd99 (DEFAULT_TABLE table=test-table [id=88180c8c57f9491fb6a7183fd4d87cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:28.614878 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.591908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:35911 (local address 127.0.0.1:44770)
0925 02:55:28.602705 (+ 10797us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:28.602745 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:28.602908 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:28.605282 (+ 2374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:28.605287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:28.605326 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:28.605870 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.605884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.607588 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.607596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.610295 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.610310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.612583 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.612590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.612767 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.612859 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 02:55:28.612944 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:55:28.614443 (+ 1499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10627,"mutex_wait_us":24,"thread_start_us":11104,"threads_started":1}
I20260925 02:55:28.615471 22606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3979dfe7e444a6a725be210205cd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:28.616724 22740 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3979dfe7e444a6a725be210205cd99 (DEFAULT_TABLE table=test-table [id=88180c8c57f9491fb6a7183fd4d87cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:28.618198 22740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3979dfe7e444a6a725be210205cd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:28.618265 22472 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3979dfe7e444a6a725be210205cd99 (DEFAULT_TABLE table=test-table [id=88180c8c57f9491fb6a7183fd4d87cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:28.619544 22472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3979dfe7e444a6a725be210205cd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:28.631377 22823 tablet_bootstrap.cc:492] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee: Bootstrap starting.
I20260925 02:55:28.633741 22824 tablet_bootstrap.cc:492] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51: Bootstrap starting.
I20260925 02:55:28.635870 22823 tablet_bootstrap.cc:654] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:28.636296 22825 tablet_bootstrap.cc:492] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6: Bootstrap starting.
I20260925 02:55:28.637358 22823 log.cc:824] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:28.637856 22824 tablet_bootstrap.cc:654] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:28.639103 22824 log.cc:824] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:28.639698 22825 tablet_bootstrap.cc:654] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:28.640909 22823 tablet_bootstrap.cc:492] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee: No bootstrap required, opened a new log
I20260925 02:55:28.640986 22825 log.cc:824] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:28.641196 22823 ts_tablet_manager.cc:1397] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 02:55:28.641772 22824 tablet_bootstrap.cc:492] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51: No bootstrap required, opened a new log
I20260925 02:55:28.642177 22824 ts_tablet_manager.cc:1397] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260925 02:55:28.646129 22825 tablet_bootstrap.cc:492] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6: No bootstrap required, opened a new log
I20260925 02:55:28.646386 22825 ts_tablet_manager.cc:1397] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260925 02:55:28.649885 22824 raft_consensus.cc:348] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 46399 } } peers { permanent_uuid: "a4815d8ac9e14606972b92d3351fb7d6" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 42213 } } peers { permanent_uuid: "14aa133b7034489f94a31c1ab5dd5a51" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35911 } }
I20260925 02:55:28.650259 22824 raft_consensus.cc:374] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:28.650372 22824 raft_consensus.cc:729] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14aa133b7034489f94a31c1ab5dd5a51, State: Initialized, Role: FOLLOWER
I20260925 02:55:28.651089 22824 consensus_queue.cc:260] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 46399 } } peers { permanent_uuid: "a4815d8ac9e14606972b92d3351fb7d6" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 42213 } } peers { permanent_uuid: "14aa133b7034489f94a31c1ab5dd5a51" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35911 } }
I20260925 02:55:28.652244 22823 raft_consensus.cc:348] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 46399 } } peers { permanent_uuid: "a4815d8ac9e14606972b92d3351fb7d6" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 42213 } } peers { permanent_uuid: "14aa133b7034489f94a31c1ab5dd5a51" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35911 } }
I20260925 02:55:28.652736 22823 raft_consensus.cc:374] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:28.652870 22823 raft_consensus.cc:729] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f99871b354444b4b8dbb18e59c1cb6ee, State: Initialized, Role: FOLLOWER
I20260925 02:55:28.653614 22823 consensus_queue.cc:260] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 46399 } } peers { permanent_uuid: "a4815d8ac9e14606972b92d3351fb7d6" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 42213 } } peers { permanent_uuid: "14aa133b7034489f94a31c1ab5dd5a51" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35911 } }
I20260925 02:55:28.654690 22824 ts_tablet_manager.cc:1428] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51: Time spent starting tablet: real 0.012s user 0.008s sys 0.002s
I20260925 02:55:28.655666 22671 heartbeater.cc:503] Master 127.2.172.126:43211 was elected leader, sending a full tablet report...
I20260925 02:55:28.656314 22823 ts_tablet_manager.cc:1428] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260925 02:55:28.657233 22825 raft_consensus.cc:348] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 46399 } } peers { permanent_uuid: "a4815d8ac9e14606972b92d3351fb7d6" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 42213 } } peers { permanent_uuid: "14aa133b7034489f94a31c1ab5dd5a51" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35911 } }
I20260925 02:55:28.657603 22825 raft_consensus.cc:374] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:28.657704 22825 raft_consensus.cc:729] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4815d8ac9e14606972b92d3351fb7d6, State: Initialized, Role: FOLLOWER
I20260925 02:55:28.658435 22825 consensus_queue.cc:260] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 46399 } } peers { permanent_uuid: "a4815d8ac9e14606972b92d3351fb7d6" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 42213 } } peers { permanent_uuid: "14aa133b7034489f94a31c1ab5dd5a51" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35911 } }
I20260925 02:55:28.660419 22825 ts_tablet_manager.cc:1428] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260925 02:55:28.660561 22805 heartbeater.cc:503] Master 127.2.172.126:43211 was elected leader, sending a full tablet report...
I20260925 02:55:28.693943 22830 raft_consensus.cc:482] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 02:55:28.694051 22538 tablet.cc:2369] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:28.694314 22830 raft_consensus.cc:504] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 46399 } } peers { permanent_uuid: "a4815d8ac9e14606972b92d3351fb7d6" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 42213 } } peers { permanent_uuid: "14aa133b7034489f94a31c1ab5dd5a51" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35911 } }
I20260925 02:55:28.696166 22830 leader_election.cc:288] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4815d8ac9e14606972b92d3351fb7d6 (127.2.172.67:42213), 14aa133b7034489f94a31c1ab5dd5a51 (127.2.172.65:35911)
I20260925 02:55:28.704521 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.696602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:37923 (local address 127.2.172.67:42213)
0925 02:55:28.696909 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:28.696919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:28.697388 (+ 469us) server_negotiation.cc:407] Connection header received
0925 02:55:28.697689 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:28.697699 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:28.697816 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:28.697934 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:28.698632 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.699749 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.702298 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.703515 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.703713 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.703811 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:28.704135 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 02:55:28.704267 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:55:28.704692 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.697689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:49799 (local address 127.2.172.65:35911)
0925 02:55:28.697908 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:28.697915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:28.698385 (+ 470us) server_negotiation.cc:407] Connection header received
0925 02:55:28.698576 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:28.698584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:28.698658 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:28.698777 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:28.699241 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.700173 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.702490 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.703724 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.703875 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.703931 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:28.704280 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 02:55:28.704443 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:55:28.704711 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.696575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:42213 (local address 127.2.172.66:37923)
0925 02:55:28.697254 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:28.697277 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:28.697484 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:28.698030 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:28.698040 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:28.698071 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:28.698491 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.698506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.699875 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.699878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.702150 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.702165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.703724 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.703729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.703922 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.703989 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 02:55:28.704089 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:55:28.704227 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":84,"thread_start_us":392,"threads_started":1}
I20260925 02:55:28.704734 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.697471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:35911 (local address 127.2.172.66:49799)
0925 02:55:28.698262 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:28.698294 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:28.698430 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:28.698797 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:28.698801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:28.698825 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:28.699096 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.699103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.700313 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.700317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.702356 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:28.702367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.703907 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.703912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:28.704129 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.704157 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:55:28.704232 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:55:28.704353 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":8,"thread_start_us":443,"threads_started":1}
I20260925 02:55:28.705734 22760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3979dfe7e444a6a725be210205cd99" candidate_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4815d8ac9e14606972b92d3351fb7d6" is_pre_election: true
I20260925 02:55:28.705724 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3979dfe7e444a6a725be210205cd99" candidate_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14aa133b7034489f94a31c1ab5dd5a51" is_pre_election: true
I20260925 02:55:28.706225 22492 raft_consensus.cc:2454] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f99871b354444b4b8dbb18e59c1cb6ee in term 0.
I20260925 02:55:28.706220 22760 raft_consensus.cc:2454] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f99871b354444b4b8dbb18e59c1cb6ee in term 0.
I20260925 02:55:28.706871 22561 leader_election.cc:302] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14aa133b7034489f94a31c1ab5dd5a51, f99871b354444b4b8dbb18e59c1cb6ee; no voters:
I20260925 02:55:28.707301 22830 raft_consensus.cc:2781] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:55:28.707459 22830 raft_consensus.cc:482] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:55:28.707587 22830 raft_consensus.cc:3037] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:28.709761 22830 raft_consensus.cc:504] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 46399 } } peers { permanent_uuid: "a4815d8ac9e14606972b92d3351fb7d6" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 42213 } } peers { permanent_uuid: "14aa133b7034489f94a31c1ab5dd5a51" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35911 } }
I20260925 02:55:28.710382 22830 leader_election.cc:288] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [CANDIDATE]: Term 1 election: Requested vote from peers a4815d8ac9e14606972b92d3351fb7d6 (127.2.172.67:42213), 14aa133b7034489f94a31c1ab5dd5a51 (127.2.172.65:35911)
I20260925 02:55:28.710772 22760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3979dfe7e444a6a725be210205cd99" candidate_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4815d8ac9e14606972b92d3351fb7d6"
I20260925 02:55:28.710825 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3979dfe7e444a6a725be210205cd99" candidate_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14aa133b7034489f94a31c1ab5dd5a51"
I20260925 02:55:28.710994 22760 raft_consensus.cc:3037] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:28.711040 22492 raft_consensus.cc:3037] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:28.713287 22760 raft_consensus.cc:2454] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f99871b354444b4b8dbb18e59c1cb6ee in term 1.
I20260925 02:55:28.713331 22492 raft_consensus.cc:2454] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f99871b354444b4b8dbb18e59c1cb6ee in term 1.
I20260925 02:55:28.713702 22561 leader_election.cc:302] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4815d8ac9e14606972b92d3351fb7d6, f99871b354444b4b8dbb18e59c1cb6ee; no voters:
I20260925 02:55:28.714006 22830 raft_consensus.cc:2781] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:28.714541 22830 raft_consensus.cc:686] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 1 LEADER]: Becoming Leader. State: Replica: f99871b354444b4b8dbb18e59c1cb6ee, State: Running, Role: LEADER
I20260925 02:55:28.715047 22830 consensus_queue.cc:237] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 46399 } } peers { permanent_uuid: "a4815d8ac9e14606972b92d3351fb7d6" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 42213 } } peers { permanent_uuid: "14aa133b7034489f94a31c1ab5dd5a51" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35911 } }
I20260925 02:55:28.719771 22347 catalog_manager.cc:5495] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee reported cstate change: term changed from 0 to 1, leader changed from <none> to f99871b354444b4b8dbb18e59c1cb6ee (127.2.172.66). New cstate: current_term: 1 leader_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99871b354444b4b8dbb18e59c1cb6ee" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4815d8ac9e14606972b92d3351fb7d6" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 42213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14aa133b7034489f94a31c1ab5dd5a51" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35911 } health_report { overall_health: UNKNOWN } } }
I20260925 02:55:28.738126 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.732571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.2.172.66:46399)
0925 02:55:28.732784 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:28.732790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:28.732839 (+ 49us) server_negotiation.cc:407] Connection header received
0925 02:55:28.733085 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:28.733090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:28.733165 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:28.733317 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:55:28.733692 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.734445 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.735931 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.736325 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.736459 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.736972 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:55:28.737460 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:55:28.737621 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:28.737726 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 02:55:28.737810 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 02:55:28.744880 22672 tablet.cc:2369] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:28.746416 22760 raft_consensus.cc:1261] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6 [term 1 FOLLOWER]: Refusing update from remote peer f99871b354444b4b8dbb18e59c1cb6ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:28.746419 22492 raft_consensus.cc:1261] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51 [term 1 FOLLOWER]: Refusing update from remote peer f99871b354444b4b8dbb18e59c1cb6ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:28.747406 22834 consensus_queue.cc:1035] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "14aa133b7034489f94a31c1ab5dd5a51" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:28.747910 22830 consensus_queue.cc:1035] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4815d8ac9e14606972b92d3351fb7d6" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 42213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:28.760106 22839 mvcc.cc:205] Tried to move back new op lower bound from 7333088988134330368 to 7333088988021338112. Current Snapshot: MvccSnapshot[applied={T|T < 7333088988134330368}]
I20260925 02:55:28.772095 22347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:28.772394 22347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:28.776599 22347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ce601bf954e4a0c8a332c44f5415d99: Sending DeleteTablet for 3 replicas of tablet 0c3979dfe7e444a6a725be210205cd99
I20260925 02:55:28.778045 22472 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c3979dfe7e444a6a725be210205cd99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:28 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:44770
I20260925 02:55:28.778162 22606 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c3979dfe7e444a6a725be210205cd99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:28 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:32890
I20260925 02:55:28.778838 22740 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c3979dfe7e444a6a725be210205cd99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:28 UTC) from {username='kudu', principal='kudu/127.2.172.126@KRBTEST.COM'} at 127.0.0.1:48942
I20260925 02:55:28.780773 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 22409
I20260925 02:55:28.802901 22849 tablet_replica.cc:330] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6: stopping tablet replica
I20260925 02:55:28.803464 22849 raft_consensus.cc:2229] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:28.803664 22848 tablet_replica.cc:330] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee: stopping tablet replica
I20260925 02:55:28.804056 22848 raft_consensus.cc:2229] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:55:28.804111 22849 raft_consensus.cc:2258] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:28.804756 22848 raft_consensus.cc:2258] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:28.805744 22847 tablet_replica.cc:330] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51: stopping tablet replica
I20260925 02:55:28.806142 22847 raft_consensus.cc:2229] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:28.806628 22847 raft_consensus.cc:2258] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:28.806898 22849 ts_tablet_manager.cc:1905] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:28.808264 22848 ts_tablet_manager.cc:1905] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:28.808890 22847 ts_tablet_manager.cc:1905] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:28.813889 22849 ts_tablet_manager.cc:1918] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:28.814061 22849 log.cc:1196] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/0c3979dfe7e444a6a725be210205cd99
I20260925 02:55:28.814595 22849 ts_tablet_manager.cc:1939] T 0c3979dfe7e444a6a725be210205cd99 P a4815d8ac9e14606972b92d3351fb7d6: Deleting consensus metadata
I20260925 02:55:28.816053 22848 ts_tablet_manager.cc:1918] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:28.816223 22848 log.cc:1196] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/0c3979dfe7e444a6a725be210205cd99
I20260925 02:55:28.816658 22847 ts_tablet_manager.cc:1918] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:28.816761 22848 ts_tablet_manager.cc:1939] T 0c3979dfe7e444a6a725be210205cd99 P f99871b354444b4b8dbb18e59c1cb6ee: Deleting consensus metadata
I20260925 02:55:28.816982 22334 catalog_manager.cc:4841] TS a4815d8ac9e14606972b92d3351fb7d6 (127.2.172.67:42213): tablet 0c3979dfe7e444a6a725be210205cd99 (table test-table [id=88180c8c57f9491fb6a7183fd4d87cc6]) successfully deleted
I20260925 02:55:28.817052 22847 log.cc:1196] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304596269475-2737-0/minicluster-data/ts-0/wal/wals/0c3979dfe7e444a6a725be210205cd99
I20260925 02:55:28.817690 22847 ts_tablet_manager.cc:1939] T 0c3979dfe7e444a6a725be210205cd99 P 14aa133b7034489f94a31c1ab5dd5a51: Deleting consensus metadata
I20260925 02:55:28.818356 22335 catalog_manager.cc:4841] TS f99871b354444b4b8dbb18e59c1cb6ee (127.2.172.66:46399): tablet 0c3979dfe7e444a6a725be210205cd99 (table test-table [id=88180c8c57f9491fb6a7183fd4d87cc6]) successfully deleted
Sep 25 02:55:28 dist-test-slave-g1xh krb5kdc[22303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.65@KRBTEST.COM
I20260925 02:55:28.819416 22334 catalog_manager.cc:4841] TS 14aa133b7034489f94a31c1ab5dd5a51 (127.2.172.65:35911): tablet 0c3979dfe7e444a6a725be210205cd99 (table test-table [id=88180c8c57f9491fb6a7183fd4d87cc6]) successfully deleted
I20260925 02:55:28.826308 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.782184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.2.172.65:35911)
0925 02:55:28.782410 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:28.782419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:28.798478 (+ 16059us) server_negotiation.cc:407] Connection header received
0925 02:55:28.799095 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:28.799105 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:28.799229 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:28.799332 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:28.801606 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.802929 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.813425 (+ 10496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.814143 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.814252 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.820296 (+ 6044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:28.820308 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:28.820322 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:28.820356 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:28.823242 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:28.824106 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:28.824113 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:28.824116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:28.824260 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:28.824730 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:28.824733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:28.824775 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:28.825132 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:28.825457 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:28.825726 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 02:55:28.825945 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:55:28.826949 22532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:28.918370 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 22409
I20260925 02:55:28.937970 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 22543
Sep 25 02:55:28 dist-test-slave-g1xh krb5kdc[22303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.66@KRBTEST.COM
I20260925 02:55:28.955638 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.938869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.2.172.66:46399)
0925 02:55:28.939081 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:28.939088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:28.939112 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:55:28.939220 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:28.939227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:28.939321 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:28.939437 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:28.940745 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.941970 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.943270 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:28.943937 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:28.944083 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:28.948377 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:28.948401 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:28.948422 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:28.948464 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:28.952603 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:28.953289 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:28.953299 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:28.953303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:28.953481 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:28.953852 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:28.953858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:28.953861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:28.954334 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:28.954652 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:28.954973 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 02:55:28.955203 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:55:28.956467 22666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:29.069962 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 22543
I20260925 02:55:29.090227 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 22677
Sep 25 02:55:29 dist-test-slave-g1xh krb5kdc[22303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.67@KRBTEST.COM
I20260925 02:55:29.106652 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:29.091066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.2.172.67:42213)
0925 02:55:29.091310 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:29.091317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:29.091340 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:55:29.091583 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:29.091589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:29.091674 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:29.091763 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:29.093267 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:29.094027 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:29.095428 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:29.095968 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:29.096069 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:29.100728 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:29.100745 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:29.100762 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:29.100805 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:29.103730 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:29.104470 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:29.104478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:29.104480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:29.104627 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:29.105047 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:29.105052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:29.105055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:29.105406 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:29.105725 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:29.106064 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 02:55:29.106308 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 02:55:29.107335 22800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:29.196522 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 22677
I20260925 02:55:29.214804 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 22316
I20260925 02:55:29.216022 22378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:29.328215 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 22316
2026-09-25T02:55:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4432 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T02:55:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:29Z Disabled control of system clock
I20260925 02:55:29.411978 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:33171
--webserver_interface=127.2.172.126
--webserver_port=0
--builtin_ntp_servers=127.2.172.84:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:33171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:55:29.754114 22858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:29.754408 22858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:29.754571 22858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:29.762070 22858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:29.762187 22858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:29.762288 22858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:29.762378 22858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:29.762449 22858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:29.772018 22858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:33171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:33171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:29.773437 22858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:29.775272 22858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:29.787365 22863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:29.787365 22866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:29.787388 22864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:29.788569 22858 server_base.cc:1034] running on GCE node
I20260925 02:55:29.789382 22858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:29.790895 22858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:29.792146 22858 hybrid_clock.cc:648] HybridClock initialized: now 1790304929792076 us; error 30 us; skew 500 ppm
I20260925 02:55:29.792547 22858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:55:29.794940 22858 webserver.cc:460] Webserver started at http://127.2.172.126:40039/ using document root <none> and password file <none>
I20260925 02:55:29.795688 22858 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:29.795809 22858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:29.796137 22858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:29.799150 22858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "7dd333f53b91492bb66a8029fbcc71a4"
format_stamp: "Formatted at 2026-09-25 02:55:29 on dist-test-slave-g1xh"
server_key: "968db3a34b1c12a4f4a0ea15c92faa25"
server_key_iv: "d904b07efc440761cd5a786611bd829c"
server_key_version: "encryptionkey@0"
I20260925 02:55:29.800060 22858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "7dd333f53b91492bb66a8029fbcc71a4"
format_stamp: "Formatted at 2026-09-25 02:55:29 on dist-test-slave-g1xh"
server_key: "968db3a34b1c12a4f4a0ea15c92faa25"
server_key_iv: "d904b07efc440761cd5a786611bd829c"
server_key_version: "encryptionkey@0"
I20260925 02:55:29.805773 22858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 02:55:29.809491 22872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:29.810978 22858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:55:29.811224 22858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "7dd333f53b91492bb66a8029fbcc71a4"
format_stamp: "Formatted at 2026-09-25 02:55:29 on dist-test-slave-g1xh"
server_key: "968db3a34b1c12a4f4a0ea15c92faa25"
server_key_iv: "d904b07efc440761cd5a786611bd829c"
server_key_version: "encryptionkey@0"
I20260925 02:55:29.811491 22858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:29.840382 22858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:29.841288 22858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:29.841583 22858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:29.861306 22858 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:33171
I20260925 02:55:29.861325 22923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:33171 every 8 connection(s)
I20260925 02:55:29.862673 22858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:29.863976 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 22858
I20260925 02:55:29.864171 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:29.864711 2737 external_mini_cluster.cc:1428] Setting key bca799896136388ede8ac03fe305800f
I20260925 02:55:29.868696 22924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:29.878397 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:29.866542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.2.172.126:33171)
0925 02:55:29.867424 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:29.867434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:29.867475 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:55:29.869337 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:29.869359 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:29.869800 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:29.870183 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:29.870740 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:29.872196 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:29.873588 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:29.874287 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:29.874494 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:29.875975 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:29.875984 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:29.875998 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:29.876555 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:29.876638 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:29.876660 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:29.876803 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:29.877051 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 02:55:29.877374 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":294,"threads_started":1}
I20260925 02:55:29.880218 22924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4: Bootstrap starting.
I20260925 02:55:29.883262 22924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:29.884310 22924 log.cc:824] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:29.887159 22924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4: No bootstrap required, opened a new log
I20260925 02:55:29.894795 22924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd333f53b91492bb66a8029fbcc71a4" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33171 } }
I20260925 02:55:29.895164 22924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:29.895290 22924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dd333f53b91492bb66a8029fbcc71a4, State: Initialized, Role: FOLLOWER
I20260925 02:55:29.895948 22924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd333f53b91492bb66a8029fbcc71a4" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33171 } }
I20260925 02:55:29.896176 22924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:29.896317 22924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:29.896497 22924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:29.898942 22924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd333f53b91492bb66a8029fbcc71a4" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33171 } }
I20260925 02:55:29.899456 22924 leader_election.cc:302] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dd333f53b91492bb66a8029fbcc71a4; no voters:
I20260925 02:55:29.900070 22924 leader_election.cc:288] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:29.900228 22929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:29.901005 22929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [term 1 LEADER]: Becoming Leader. State: Replica: 7dd333f53b91492bb66a8029fbcc71a4, State: Running, Role: LEADER
I20260925 02:55:29.901612 22929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd333f53b91492bb66a8029fbcc71a4" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33171 } }
I20260925 02:55:29.902269 22924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:29.904488 22931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dd333f53b91492bb66a8029fbcc71a4. Latest consensus state: current_term: 1 leader_uuid: "7dd333f53b91492bb66a8029fbcc71a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd333f53b91492bb66a8029fbcc71a4" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33171 } } }
I20260925 02:55:29.904938 22931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:29.904585 22930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dd333f53b91492bb66a8029fbcc71a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd333f53b91492bb66a8029fbcc71a4" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 33171 } } }
I20260925 02:55:29.905162 22930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:29.908608 22934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:29.913717 22934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:29.924340 22934 catalog_manager.cc:1347] Generated new cluster ID: fa8ceeba7f83452aa215b221b8322fb4
I20260925 02:55:29.924496 22934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:29.949977 22934 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:29.952051 22934 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:29.969807 22934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4: Generated new TSK 0
I20260925 02:55:29.970980 22934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:29.980607 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.65:0
--local_ip_for_outbound_sockets=127.2.172.65
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:33171
--builtin_ntp_servers=127.2.172.84:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:55:30.329301 22948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:30.329588 22948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:30.329675 22948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:30.337023 22948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:30.337136 22948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:30.337244 22948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.65
I20260925 02:55:30.347618 22948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.172.65
--webserver_port=0
--tserver_master_addrs=127.2.172.126:33171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.65
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:30.348922 22948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:30.350620 22948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:30.364260 22953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:30.364229 22956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:30.364228 22954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:30.365334 22948 server_base.cc:1034] running on GCE node
I20260925 02:55:30.365921 22948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:30.366873 22948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:30.368083 22948 hybrid_clock.cc:648] HybridClock initialized: now 1790304930368028 us; error 41 us; skew 500 ppm
I20260925 02:55:30.368415 22948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:55:30.376649 22948 webserver.cc:460] Webserver started at http://127.2.172.65:41751/ using document root <none> and password file <none>
I20260925 02:55:30.377463 22948 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:30.377609 22948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:30.377959 22948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:30.381108 22948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/data/instance:
uuid: "9b64dfcf45654333bef7bc36bbc0afb8"
format_stamp: "Formatted at 2026-09-25 02:55:30 on dist-test-slave-g1xh"
server_key: "d24843fedc4683178c7724213e71eb50"
server_key_iv: "636ebf3da4b9bd509a219ae1be246285"
server_key_version: "encryptionkey@0"
I20260925 02:55:30.381923 22948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance:
uuid: "9b64dfcf45654333bef7bc36bbc0afb8"
format_stamp: "Formatted at 2026-09-25 02:55:30 on dist-test-slave-g1xh"
server_key: "d24843fedc4683178c7724213e71eb50"
server_key_iv: "636ebf3da4b9bd509a219ae1be246285"
server_key_version: "encryptionkey@0"
I20260925 02:55:30.387126 22948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 02:55:30.392168 22962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:30.394090 22948 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:55:30.394277 22948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/wal
uuid: "9b64dfcf45654333bef7bc36bbc0afb8"
format_stamp: "Formatted at 2026-09-25 02:55:30 on dist-test-slave-g1xh"
server_key: "d24843fedc4683178c7724213e71eb50"
server_key_iv: "636ebf3da4b9bd509a219ae1be246285"
server_key_version: "encryptionkey@0"
I20260925 02:55:30.394567 22948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:30.407896 22948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:30.408689 22948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:30.408962 22948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:30.409855 22948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:30.411267 22948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:30.411355 22948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:30.411437 22948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:30.411490 22948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:30.445400 22948 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.65:45991
I20260925 02:55:30.445483 23074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.65:45991 every 8 connection(s)
I20260925 02:55:30.446831 22948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:30.448271 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 22948
I20260925 02:55:30.448478 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-0/wal/instance
I20260925 02:55:30.448992 2737 external_mini_cluster.cc:1428] Setting key f86269d4f66ca93da65d0e0b145bc17a
I20260925 02:55:30.452250 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.66:0
--local_ip_for_outbound_sockets=127.2.172.66
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:33171
--builtin_ntp_servers=127.2.172.84:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:55:30.462639 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:30.449641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.65:58757 (local address 127.2.172.126:33171)
0925 02:55:30.450150 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:30.450156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:30.451517 (+ 1361us) server_negotiation.cc:407] Connection header received
0925 02:55:30.454429 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:30.454433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:30.454504 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:30.454595 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:30.456133 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:30.456853 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:30.459489 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:30.459997 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:30.460112 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:30.461855 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:30.461860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:30.461863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:30.461883 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:30.461939 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:30.461952 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:30.462027 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:30.462294 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 02:55:30.462381 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":276,"threads_started":1}
I20260925 02:55:30.463901 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:30.450263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33171 (local address 127.2.172.65:58757)
0925 02:55:30.451346 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:30.451401 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:30.454089 (+ 2688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:30.454900 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:30.454915 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:30.455052 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:30.455943 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:30.455972 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:30.457026 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:30.457033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:30.459286 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:30.459307 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:30.460260 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:30.460279 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:30.460634 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:30.461458 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:55:30.461464 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:30.461523 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:55:30.461634 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:30.462057 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:30.462071 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:30.462082 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 02:55:30.462289 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 02:55:30.462745 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":318,"threads_started":1}
I20260925 02:55:30.465608 23075 heartbeater.cc:347] Connected to a master server at 127.2.172.126:33171
I20260925 02:55:30.465925 23075 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:30.466733 23075 heartbeater.cc:511] Master 127.2.172.126:33171 requested a full tablet report, sending...
I20260925 02:55:30.469105 22889 ts_manager.cc:194] Registered new tserver with Master: 9b64dfcf45654333bef7bc36bbc0afb8 (127.2.172.65:45991)
I20260925 02:55:30.471366 22889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.172.65:58757
W20260925 02:55:30.795228 23079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:30.795531 23079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:30.795725 23079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:30.803328 23079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:30.803427 23079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:30.803534 23079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.66
I20260925 02:55:30.813027 23079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.172.66
--webserver_port=0
--tserver_master_addrs=127.2.172.126:33171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.66
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:30.814373 23079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:30.816427 23079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:30.829460 23085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:30.829535 23084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:30.829514 23087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:30.830324 23079 server_base.cc:1034] running on GCE node
I20260925 02:55:30.830904 23079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:30.831881 23079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:30.833106 23079 hybrid_clock.cc:648] HybridClock initialized: now 1790304930833073 us; error 45 us; skew 500 ppm
I20260925 02:55:30.833492 23079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:55:30.835893 23079 webserver.cc:460] Webserver started at http://127.2.172.66:39095/ using document root <none> and password file <none>
I20260925 02:55:30.836593 23079 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:30.836730 23079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:30.837054 23079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:30.840106 23079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/data/instance:
uuid: "348fe4e6b50c4ae2967fa7c9b745398a"
format_stamp: "Formatted at 2026-09-25 02:55:30 on dist-test-slave-g1xh"
server_key: "aa0e8d27a8e40b38a197e22d699c21cd"
server_key_iv: "7a98a6ec6363952c186ecf4a5e719553"
server_key_version: "encryptionkey@0"
I20260925 02:55:30.840895 23079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance:
uuid: "348fe4e6b50c4ae2967fa7c9b745398a"
format_stamp: "Formatted at 2026-09-25 02:55:30 on dist-test-slave-g1xh"
server_key: "aa0e8d27a8e40b38a197e22d699c21cd"
server_key_iv: "7a98a6ec6363952c186ecf4a5e719553"
server_key_version: "encryptionkey@0"
I20260925 02:55:30.845988 23079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:55:30.849493 23093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:30.850843 23079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:55:30.851022 23079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/wal
uuid: "348fe4e6b50c4ae2967fa7c9b745398a"
format_stamp: "Formatted at 2026-09-25 02:55:30 on dist-test-slave-g1xh"
server_key: "aa0e8d27a8e40b38a197e22d699c21cd"
server_key_iv: "7a98a6ec6363952c186ecf4a5e719553"
server_key_version: "encryptionkey@0"
I20260925 02:55:30.851253 23079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:30.873386 23079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:30.874289 23079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:30.874593 23079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:30.875638 23079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:30.877077 23079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:30.877209 23079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:30.877347 23079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:30.877437 23079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:30.909716 23079 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.66:37977
I20260925 02:55:30.909806 23205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.66:37977 every 8 connection(s)
I20260925 02:55:30.911016 23079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:30.915227 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 23079
I20260925 02:55:30.915424 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/wal/instance
I20260925 02:55:30.916021 2737 external_mini_cluster.cc:1428] Setting key 8024a70d82ce21128bbdc80743b60be7
I20260925 02:55:30.919245 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.172.67:0
--local_ip_for_outbound_sockets=127.2.172.67
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:33171
--builtin_ntp_servers=127.2.172.84:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:55:30.925748 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:30.913685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.66:49673 (local address 127.2.172.126:33171)
0925 02:55:30.913916 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:30.913923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:30.915653 (+ 1730us) server_negotiation.cc:407] Connection header received
0925 02:55:30.917980 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:30.917985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:30.918067 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:30.918174 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:30.919666 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:30.920405 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:30.922865 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:30.923233 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:30.923326 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:30.924961 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:30.924966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:30.924969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:30.924988 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:30.925020 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:30.925032 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:30.925099 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:30.925430 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 02:55:30.925516 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 02:55:30.927181 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:30.914185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33171 (local address 127.2.172.66:49673)
0925 02:55:30.915441 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:30.915495 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:30.917658 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:30.918459 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:30.918471 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:30.918592 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:30.919469 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:30.919498 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:30.920563 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:30.920572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:30.922718 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:30.922738 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:30.923459 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:30.923467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:30.923830 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:30.924583 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:55:30.924588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:30.924649 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:55:30.924760 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:30.925130 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:30.925143 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:30.925154 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 02:55:30.925345 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 02:55:30.925785 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":307,"threads_started":1}
I20260925 02:55:30.928718 23206 heartbeater.cc:347] Connected to a master server at 127.2.172.126:33171
I20260925 02:55:30.929051 23206 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:30.929888 23206 heartbeater.cc:511] Master 127.2.172.126:33171 requested a full tablet report, sending...
I20260925 02:55:30.931354 22889 ts_manager.cc:194] Registered new tserver with Master: 348fe4e6b50c4ae2967fa7c9b745398a (127.2.172.66:37977)
I20260925 02:55:30.932415 22889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.172.66:49673
W20260925 02:55:31.257586 23209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:31.257884 23209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:31.258049 23209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:31.265457 23209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:31.265582 23209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:31.265771 23209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.172.67
I20260925 02:55:31.274968 23209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.172.67
--webserver_port=0
--tserver_master_addrs=127.2.172.126:33171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.172.67
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:31.276645 23209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:31.278571 23209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:31.291796 23215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:31.291813 23214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:31.292407 23217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:31.292596 23209 server_base.cc:1034] running on GCE node
I20260925 02:55:31.293233 23209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:31.294195 23209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:31.295433 23209 hybrid_clock.cc:648] HybridClock initialized: now 1790304931295393 us; error 53 us; skew 500 ppm
I20260925 02:55:31.295789 23209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:55:31.298206 23209 webserver.cc:460] Webserver started at http://127.2.172.67:34111/ using document root <none> and password file <none>
I20260925 02:55:31.299063 23209 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:31.299199 23209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:31.299532 23209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:31.302608 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/data/instance:
uuid: "c603de8386154bb6adfd36285111154d"
format_stamp: "Formatted at 2026-09-25 02:55:31 on dist-test-slave-g1xh"
server_key: "9a34b2a3775462c7a458f5c8f1971aff"
server_key_iv: "d389ecd3c092ba11d40a52ce071b9006"
server_key_version: "encryptionkey@0"
I20260925 02:55:31.303460 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance:
uuid: "c603de8386154bb6adfd36285111154d"
format_stamp: "Formatted at 2026-09-25 02:55:31 on dist-test-slave-g1xh"
server_key: "9a34b2a3775462c7a458f5c8f1971aff"
server_key_iv: "d389ecd3c092ba11d40a52ce071b9006"
server_key_version: "encryptionkey@0"
I20260925 02:55:31.308481 23209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 02:55:31.311967 23223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:31.313419 23209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:55:31.313621 23209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/wal
uuid: "c603de8386154bb6adfd36285111154d"
format_stamp: "Formatted at 2026-09-25 02:55:31 on dist-test-slave-g1xh"
server_key: "9a34b2a3775462c7a458f5c8f1971aff"
server_key_iv: "d389ecd3c092ba11d40a52ce071b9006"
server_key_version: "encryptionkey@0"
I20260925 02:55:31.313858 23209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:31.338135 23209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:31.339057 23209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:31.339404 23209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:31.340653 23209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:31.342207 23209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:31.342342 23209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:31.342480 23209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:31.342571 23209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:31.377401 23209 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.67:45365
I20260925 02:55:31.377501 23335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.67:45365 every 8 connection(s)
I20260925 02:55:31.378782 23209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/data/info.pb
I20260925 02:55:31.380015 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 23209
I20260925 02:55:31.380200 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/wal/instance
I20260925 02:55:31.380786 2737 external_mini_cluster.cc:1428] Setting key b01e98895d7e48ed8e72dfe2dbbd30d5
I20260925 02:55:31.392346 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.381632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:34089 (local address 127.2.172.126:33171)
0925 02:55:31.381830 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:31.381836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:31.383588 (+ 1752us) server_negotiation.cc:407] Connection header received
0925 02:55:31.385524 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:31.385528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:31.385597 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:31.385732 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:31.386800 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.387792 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.389357 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.389739 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.389838 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.391324 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:31.391330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:31.391333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:31.391352 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:31.391388 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:31.391401 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:31.391476 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:31.391999 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 02:55:31.392076 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:55:31.393362 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.382222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:33171 (local address 127.2.172.67:34089)
0925 02:55:31.383391 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.383459 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:31.385254 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:31.385874 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:31.385881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:31.385975 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:31.386537 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:31.386552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.387973 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.387979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:31.389223 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:31.389235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.389937 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.389942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:31.390145 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.390820 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:55:31.390823 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:31.390862 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:55:31.391117 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:31.391514 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:31.391522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:31.391529 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:55:31.391972 (+ 443us) client_negotiation.cc:240] Negotiation successful
0925 02:55:31.392267 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":446,"threads_started":1}
I20260925 02:55:31.394868 23336 heartbeater.cc:347] Connected to a master server at 127.2.172.126:33171
I20260925 02:55:31.395231 23336 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:31.396220 23336 heartbeater.cc:511] Master 127.2.172.126:33171 requested a full tablet report, sending...
I20260925 02:55:31.397650 22889 ts_manager.cc:194] Registered new tserver with Master: c603de8386154bb6adfd36285111154d (127.2.172.67:45365)
I20260925 02:55:31.398522 22889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.172.67:34089
I20260925 02:55:31.408871 2737 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:55:31.419540 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.413132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.2.172.126:33171)
0925 02:55:31.413312 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:31.413320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:31.413752 (+ 432us) server_negotiation.cc:407] Connection header received
0925 02:55:31.413953 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:31.413958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:31.414049 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:31.414159 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:31.414754 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.415663 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.417104 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.417582 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.417728 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.418837 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:31.418843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:31.418849 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:31.418872 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:31.418901 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:31.418913 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:31.419007 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:31.419156 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 02:55:31.419246 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:55:31.423131 22889 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:55:31.430063 22889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:55:31.454221 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.442933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45991 (local address 127.0.0.1:52150)
0925 02:55:31.444640 (+ 1707us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.444657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:31.444770 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:31.445844 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:31.445853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:31.445887 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:31.446312 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:31.446328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.448732 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.448738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:31.451245 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:31.451261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.453296 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.453303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:31.453498 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.453553 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 02:55:31.453660 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:55:31.453853 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1575,"mutex_wait_us":63,"thread_start_us":555,"threads_started":1}
I20260925 02:55:31.455334 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.442044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.67:45365 (local address 127.0.0.1:59186)
0925 02:55:31.443888 (+ 1844us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.443938 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:31.444091 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:31.445449 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:31.445456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:31.445485 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:31.445937 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:31.445955 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.449350 (+ 3395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.449355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:31.451490 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:31.451502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.454530 (+ 3028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.454537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:31.454739 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.454789 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 02:55:31.454852 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 02:55:31.455000 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1612,"thread_start_us":2333,"threads_started":1}
I20260925 02:55:31.456519 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.443278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.2.172.65:45991)
0925 02:55:31.445241 (+ 1963us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:31.445251 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:31.445295 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:55:31.445399 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:31.445406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:31.445622 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:31.446516 (+ 894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:31.446785 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.448578 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.451508 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.453077 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.455157 (+ 2080us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.455563 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:31.455647 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:55:31.455882 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1820,"thread_start_us":8289,"threads_started":1}
I20260925 02:55:31.456872 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.442247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.2.172.67:45365)
0925 02:55:31.444671 (+ 2424us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:31.444684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:31.444720 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:55:31.444955 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:31.444966 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:31.445216 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:31.446163 (+ 947us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:31.447516 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.449192 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.452882 (+ 3690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.454306 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.455617 (+ 1311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.456253 (+ 636us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:31.456330 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 02:55:31.456513 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2246,"thread_start_us":500,"threads_started":1}
I20260925 02:55:31.460407 23271 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1a51e684a449bf9517e40a394847a8 (DEFAULT_TABLE table=test-table [id=7b7446c240a0419a8bf3dcf7809ec234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:31.460389 23010 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1a51e684a449bf9517e40a394847a8 (DEFAULT_TABLE table=test-table [id=7b7446c240a0419a8bf3dcf7809ec234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:31.461947 23010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1a51e684a449bf9517e40a394847a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:31.462245 23271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1a51e684a449bf9517e40a394847a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:31.464656 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.445243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:37977 (local address 127.0.0.1:37814)
0925 02:55:31.454536 (+ 9293us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.454556 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:31.454659 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:31.458031 (+ 3372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:31.458037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:31.458065 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:31.458215 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:31.458226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.460112 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.460118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:31.461991 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:31.462009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.463826 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.463832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:31.464032 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.464066 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:55:31.464171 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 02:55:31.464327 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9178,"thread_start_us":10482,"threads_started":1}
I20260925 02:55:31.464740 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.445002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.2.172.66:37977)
0925 02:55:31.456249 (+ 11247us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:31.456257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:31.456291 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:55:31.456417 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:31.456426 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:31.457793 (+ 1367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:31.458023 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:31.458391 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.459962 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.462195 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.463644 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.463791 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.464049 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:31.464229 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 02:55:31.464412 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11095,"thread_start_us":11006,"threads_started":1}
I20260925 02:55:31.469027 23141 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1a51e684a449bf9517e40a394847a8 (DEFAULT_TABLE table=test-table [id=7b7446c240a0419a8bf3dcf7809ec234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:31.470502 23141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1a51e684a449bf9517e40a394847a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:31.474509 23075 heartbeater.cc:503] Master 127.2.172.126:33171 was elected leader, sending a full tablet report...
I20260925 02:55:31.475471 23354 tablet_bootstrap.cc:492] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8: Bootstrap starting.
I20260925 02:55:31.479403 23354 tablet_bootstrap.cc:654] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:31.479390 23355 tablet_bootstrap.cc:492] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d: Bootstrap starting.
I20260925 02:55:31.480942 23354 log.cc:824] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:31.483095 23355 tablet_bootstrap.cc:654] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:31.484494 23354 tablet_bootstrap.cc:492] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8: No bootstrap required, opened a new log
I20260925 02:55:31.484441 23355 log.cc:824] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:31.484926 23354 ts_tablet_manager.cc:1397] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260925 02:55:31.487407 23358 tablet_bootstrap.cc:492] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a: Bootstrap starting.
I20260925 02:55:31.490947 23358 tablet_bootstrap.cc:654] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:31.491324 23355 tablet_bootstrap.cc:492] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d: No bootstrap required, opened a new log
I20260925 02:55:31.491653 23355 ts_tablet_manager.cc:1397] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260925 02:55:31.492175 23358 log.cc:824] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:31.495052 23358 tablet_bootstrap.cc:492] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a: No bootstrap required, opened a new log
I20260925 02:55:31.495846 23358 ts_tablet_manager.cc:1397] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 02:55:31.496348 23354 raft_consensus.cc:348] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c603de8386154bb6adfd36285111154d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 45365 } } peers { permanent_uuid: "348fe4e6b50c4ae2967fa7c9b745398a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37977 } } peers { permanent_uuid: "9b64dfcf45654333bef7bc36bbc0afb8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45991 } }
I20260925 02:55:31.496821 23354 raft_consensus.cc:374] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:31.496970 23354 raft_consensus.cc:729] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b64dfcf45654333bef7bc36bbc0afb8, State: Initialized, Role: FOLLOWER
I20260925 02:55:31.497635 23354 consensus_queue.cc:260] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c603de8386154bb6adfd36285111154d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 45365 } } peers { permanent_uuid: "348fe4e6b50c4ae2967fa7c9b745398a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37977 } } peers { permanent_uuid: "9b64dfcf45654333bef7bc36bbc0afb8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45991 } }
I20260925 02:55:31.499632 23354 ts_tablet_manager.cc:1428] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260925 02:55:31.501686 23355 raft_consensus.cc:348] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c603de8386154bb6adfd36285111154d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 45365 } } peers { permanent_uuid: "348fe4e6b50c4ae2967fa7c9b745398a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37977 } } peers { permanent_uuid: "9b64dfcf45654333bef7bc36bbc0afb8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45991 } }
I20260925 02:55:31.502152 23355 raft_consensus.cc:374] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:31.502259 23355 raft_consensus.cc:729] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c603de8386154bb6adfd36285111154d, State: Initialized, Role: FOLLOWER
I20260925 02:55:31.503119 23355 consensus_queue.cc:260] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c603de8386154bb6adfd36285111154d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 45365 } } peers { permanent_uuid: "348fe4e6b50c4ae2967fa7c9b745398a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37977 } } peers { permanent_uuid: "9b64dfcf45654333bef7bc36bbc0afb8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45991 } }
I20260925 02:55:31.504359 23336 heartbeater.cc:503] Master 127.2.172.126:33171 was elected leader, sending a full tablet report...
I20260925 02:55:31.504920 23355 ts_tablet_manager.cc:1428] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260925 02:55:31.507179 23358 raft_consensus.cc:348] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c603de8386154bb6adfd36285111154d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 45365 } } peers { permanent_uuid: "348fe4e6b50c4ae2967fa7c9b745398a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37977 } } peers { permanent_uuid: "9b64dfcf45654333bef7bc36bbc0afb8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45991 } }
I20260925 02:55:31.507694 23358 raft_consensus.cc:374] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:31.507814 23358 raft_consensus.cc:729] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348fe4e6b50c4ae2967fa7c9b745398a, State: Initialized, Role: FOLLOWER
I20260925 02:55:31.508491 23358 consensus_queue.cc:260] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c603de8386154bb6adfd36285111154d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 45365 } } peers { permanent_uuid: "348fe4e6b50c4ae2967fa7c9b745398a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37977 } } peers { permanent_uuid: "9b64dfcf45654333bef7bc36bbc0afb8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45991 } }
I20260925 02:55:31.509766 23206 heartbeater.cc:503] Master 127.2.172.126:33171 was elected leader, sending a full tablet report...
I20260925 02:55:31.510324 23358 ts_tablet_manager.cc:1428] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260925 02:55:31.550612 23361 raft_consensus.cc:482] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:31.551011 23361 raft_consensus.cc:504] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c603de8386154bb6adfd36285111154d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 45365 } } peers { permanent_uuid: "348fe4e6b50c4ae2967fa7c9b745398a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37977 } } peers { permanent_uuid: "9b64dfcf45654333bef7bc36bbc0afb8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45991 } }
I20260925 02:55:31.553485 23361 leader_election.cc:288] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 348fe4e6b50c4ae2967fa7c9b745398a (127.2.172.66:37977), 9b64dfcf45654333bef7bc36bbc0afb8 (127.2.172.65:45991)
I20260925 02:55:31.559998 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.554288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:49599 (local address 127.2.172.66:37977)
0925 02:55:31.554530 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:31.554537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:31.554557 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:55:31.554634 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:31.554642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:31.554760 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:31.554866 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:31.555582 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.556638 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.558303 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.559056 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.559212 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.559283 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:31.559616 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 02:55:31.559735 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:55:31.559998 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.553780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.66:37977 (local address 127.2.172.67:49599)
0925 02:55:31.553986 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.554007 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:31.554136 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:31.555057 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:31.555063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:31.555099 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:31.555331 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:31.555350 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.556785 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.556790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:31.558136 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:31.558147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.559212 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.559218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:31.559420 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.559443 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:55:31.559542 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 02:55:31.559692 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 02:55:31.560953 23161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1a51e684a449bf9517e40a394847a8" candidate_uuid: "c603de8386154bb6adfd36285111154d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348fe4e6b50c4ae2967fa7c9b745398a" is_pre_election: true
I20260925 02:55:31.561225 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.553780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45991 (local address 127.2.172.67:56603)
0925 02:55:31.555235 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.555249 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:31.555524 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:31.556249 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:31.556254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:31.556281 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:31.556692 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:31.556712 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.557756 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.557761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:31.559073 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:31.559084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.560683 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.560687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:31.560804 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.560826 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:55:31.560909 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:55:31.560991 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1323,"mutex_wait_us":750,"thread_start_us":375,"threads_started":1}
I20260925 02:55:31.561357 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.554454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.172.67:56603 (local address 127.2.172.65:45991)
0925 02:55:31.554644 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:31.554652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:31.555355 (+ 703us) server_negotiation.cc:407] Connection header received
0925 02:55:31.555816 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:31.555822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:31.555976 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:31.556104 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:31.556875 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.557596 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.559253 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.560111 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.560292 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.560354 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:31.560966 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 02:55:31.561108 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:55:31.561560 23161 raft_consensus.cc:2454] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c603de8386154bb6adfd36285111154d in term 0.
I20260925 02:55:31.562302 23226 leader_election.cc:302] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348fe4e6b50c4ae2967fa7c9b745398a, c603de8386154bb6adfd36285111154d; no voters:
I20260925 02:55:31.562309 23030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1a51e684a449bf9517e40a394847a8" candidate_uuid: "c603de8386154bb6adfd36285111154d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b64dfcf45654333bef7bc36bbc0afb8" is_pre_election: true
I20260925 02:55:31.562753 23361 raft_consensus.cc:2781] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:55:31.562915 23361 raft_consensus.cc:482] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:55:31.562875 23030 raft_consensus.cc:2454] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c603de8386154bb6adfd36285111154d in term 0.
I20260925 02:55:31.563025 23361 raft_consensus.cc:3037] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:31.565394 23361 raft_consensus.cc:504] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c603de8386154bb6adfd36285111154d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 45365 } } peers { permanent_uuid: "348fe4e6b50c4ae2967fa7c9b745398a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37977 } } peers { permanent_uuid: "9b64dfcf45654333bef7bc36bbc0afb8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45991 } }
I20260925 02:55:31.566015 23361 leader_election.cc:288] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [CANDIDATE]: Term 1 election: Requested vote from peers 348fe4e6b50c4ae2967fa7c9b745398a (127.2.172.66:37977), 9b64dfcf45654333bef7bc36bbc0afb8 (127.2.172.65:45991)
I20260925 02:55:31.566537 23030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1a51e684a449bf9517e40a394847a8" candidate_uuid: "c603de8386154bb6adfd36285111154d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b64dfcf45654333bef7bc36bbc0afb8"
I20260925 02:55:31.566529 23161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1a51e684a449bf9517e40a394847a8" candidate_uuid: "c603de8386154bb6adfd36285111154d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348fe4e6b50c4ae2967fa7c9b745398a"
I20260925 02:55:31.566788 23030 raft_consensus.cc:3037] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:31.566776 23161 raft_consensus.cc:3037] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:31.569546 23030 raft_consensus.cc:2454] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c603de8386154bb6adfd36285111154d in term 1.
I20260925 02:55:31.569604 23161 raft_consensus.cc:2454] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c603de8386154bb6adfd36285111154d in term 1.
I20260925 02:55:31.570034 23225 leader_election.cc:302] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b64dfcf45654333bef7bc36bbc0afb8, c603de8386154bb6adfd36285111154d; no voters:
I20260925 02:55:31.570421 23361 raft_consensus.cc:2781] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:31.570883 23361 raft_consensus.cc:686] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 1 LEADER]: Becoming Leader. State: Replica: c603de8386154bb6adfd36285111154d, State: Running, Role: LEADER
I20260925 02:55:31.571446 23361 consensus_queue.cc:237] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c603de8386154bb6adfd36285111154d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 45365 } } peers { permanent_uuid: "348fe4e6b50c4ae2967fa7c9b745398a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37977 } } peers { permanent_uuid: "9b64dfcf45654333bef7bc36bbc0afb8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45991 } }
I20260925 02:55:31.576260 22889 catalog_manager.cc:5495] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d reported cstate change: term changed from 0 to 1, leader changed from <none> to c603de8386154bb6adfd36285111154d (127.2.172.67). New cstate: current_term: 1 leader_uuid: "c603de8386154bb6adfd36285111154d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c603de8386154bb6adfd36285111154d" member_type: VOTER last_known_addr { host: "127.2.172.67" port: 45365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "348fe4e6b50c4ae2967fa7c9b745398a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b64dfcf45654333bef7bc36bbc0afb8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45991 } health_report { overall_health: UNKNOWN } } }
I20260925 02:55:31.611435 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.605980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.2.172.67:45365)
0925 02:55:31.606167 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:31.606172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:31.606275 (+ 103us) server_negotiation.cc:407] Connection header received
0925 02:55:31.606459 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:31.606463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:31.606551 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:31.606720 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:55:31.607070 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.607904 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.609215 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.609655 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.609793 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.610260 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:55:31.610862 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:55:31.610989 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:31.611078 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 02:55:31.611141 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:55:31.620262 23161 raft_consensus.cc:1261] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a [term 1 FOLLOWER]: Refusing update from remote peer c603de8386154bb6adfd36285111154d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:31.620267 23030 raft_consensus.cc:1261] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8 [term 1 FOLLOWER]: Refusing update from remote peer c603de8386154bb6adfd36285111154d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:31.621233 23364 consensus_queue.cc:1035] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b64dfcf45654333bef7bc36bbc0afb8" member_type: VOTER last_known_addr { host: "127.2.172.65" port: 45991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:31.621626 23361 consensus_queue.cc:1035] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [LEADER]: Connected to new peer: Peer: permanent_uuid: "348fe4e6b50c4ae2967fa7c9b745398a" member_type: VOTER last_known_addr { host: "127.2.172.66" port: 37977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:55:31.631057 23337 tablet.cc:2369] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:31.633227 23367 mvcc.cc:205] Tried to move back new op lower bound from 7333088999906029568 to 7333088999721414656. Current Snapshot: MvccSnapshot[applied={T|T < 7333088999906029568}]
I20260925 02:55:31.634994 23369 mvcc.cc:205] Tried to move back new op lower bound from 7333088999906029568 to 7333088999721414656. Current Snapshot: MvccSnapshot[applied={T|T < 7333088999906029568}]
I20260925 02:55:31.648068 22888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:31.648412 22888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:31.652320 22888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dd333f53b91492bb66a8029fbcc71a4: Sending DeleteTablet for 3 replicas of tablet 2e1a51e684a449bf9517e40a394847a8
I20260925 02:55:31.653633 23010 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e1a51e684a449bf9517e40a394847a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:31 UTC) from {username='slave'} at 127.0.0.1:52150
I20260925 02:55:31.653663 23141 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e1a51e684a449bf9517e40a394847a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:31 UTC) from {username='slave'} at 127.0.0.1:37814
I20260925 02:55:31.653659 23271 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e1a51e684a449bf9517e40a394847a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:31 UTC) from {username='slave'} at 127.0.0.1:59186
I20260925 02:55:31.655738 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 22948
I20260925 02:55:31.667691 23378 tablet_replica.cc:330] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d: stopping tablet replica
I20260925 02:55:31.667786 23377 tablet_replica.cc:330] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a: stopping tablet replica
I20260925 02:55:31.668248 23377 raft_consensus.cc:2229] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:31.668268 23378 raft_consensus.cc:2229] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:55:31.668953 23377 raft_consensus.cc:2258] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:31.668967 23378 raft_consensus.cc:2258] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:31.669795 23379 tablet_replica.cc:330] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8: stopping tablet replica
I20260925 02:55:31.671640 23377 ts_tablet_manager.cc:1905] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:31.671963 23378 ts_tablet_manager.cc:1905] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:31.674288 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.656948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.2.172.65:45991)
0925 02:55:31.657133 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:31.657139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:31.664301 (+ 7162us) server_negotiation.cc:407] Connection header received
0925 02:55:31.664559 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:31.664564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:31.664714 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:31.664849 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:31.666684 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.667897 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.670022 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.670693 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.670816 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.672491 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:31.672503 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:31.672521 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:31.673306 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:31.673369 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:31.673395 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:31.673482 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:31.673773 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 02:55:31.673907 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:55:31.677338 23379 raft_consensus.cc:2229] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:31.677911 23379 raft_consensus.cc:2258] T 2e1a51e684a449bf9517e40a394847a8 P 9b64dfcf45654333bef7bc36bbc0afb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:31.678156 23070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:31.679359 23378 ts_tablet_manager.cc:1918] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:31.679351 23377 ts_tablet_manager.cc:1918] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:31.679535 23377 log.cc:1196] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-1/wal/wals/2e1a51e684a449bf9517e40a394847a8
I20260925 02:55:31.679615 23378 log.cc:1196] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d: Deleting WAL directory at /tmp/dist-test-task450sZB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304596269475-2737-0/minicluster-data/ts-2/wal/wals/2e1a51e684a449bf9517e40a394847a8
I20260925 02:55:31.680197 23378 ts_tablet_manager.cc:1939] T 2e1a51e684a449bf9517e40a394847a8 P c603de8386154bb6adfd36285111154d: Deleting consensus metadata
I20260925 02:55:31.680785 23377 ts_tablet_manager.cc:1939] T 2e1a51e684a449bf9517e40a394847a8 P 348fe4e6b50c4ae2967fa7c9b745398a: Deleting consensus metadata
I20260925 02:55:31.682248 22875 catalog_manager.cc:4841] TS c603de8386154bb6adfd36285111154d (127.2.172.67:45365): tablet 2e1a51e684a449bf9517e40a394847a8 (table test-table [id=7b7446c240a0419a8bf3dcf7809ec234]) successfully deleted
I20260925 02:55:31.682627 22875 catalog_manager.cc:4841] TS 348fe4e6b50c4ae2967fa7c9b745398a (127.2.172.66:37977): tablet 2e1a51e684a449bf9517e40a394847a8 (table test-table [id=7b7446c240a0419a8bf3dcf7809ec234]) successfully deleted
I20260925 02:55:31.793174 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 22948
W20260925 02:55:31.812789 22874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.172.65:45991 (error 108)
I20260925 02:55:31.813004 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 23079
I20260925 02:55:31.816027 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.815463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45991 (local address 127.0.0.1:52176)
0925 02:55:31.815692 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.815799 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 02:55:31.816308 22874 catalog_manager.cc:4584] TS 9b64dfcf45654333bef7bc36bbc0afb8 (127.2.172.65:45991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e1a51e684a449bf9517e40a394847a8: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45991: connect: Connection refused (error 111)
I20260925 02:55:31.821805 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.814319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.2.172.66:37977)
0925 02:55:31.814533 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:31.814539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:31.814576 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:55:31.814652 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:31.814667 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:31.814799 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:31.814941 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:31.815450 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.816658 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.818103 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.818822 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.818948 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.820169 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:31.820181 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:31.820199 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:31.820991 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:31.821053 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:31.821079 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:31.821200 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:31.821328 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 02:55:31.821445 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:55:31.822515 23201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:31.869920 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.869456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45991 (local address 127.0.0.1:52186)
0925 02:55:31.869632 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.869731 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 02:55:31.871147 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.870783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45991 (local address 127.0.0.1:52200)
0925 02:55:31.870909 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.870979 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:55:31.927488 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 23079
I20260925 02:55:31.931977 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.931346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45991 (local address 127.0.0.1:52210)
0925 02:55:31.931654 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.931773 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 02:55:31.933228 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.932757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45991 (local address 127.0.0.1:52218)
0925 02:55:31.932943 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.933043 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 02:55:31.947024 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 23209
I20260925 02:55:31.955096 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.947967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.2.172.67:45365)
0925 02:55:31.948145 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:31.948149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:31.948172 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:55:31.948431 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:31.948435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:31.948515 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:31.948598 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:31.949095 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.949902 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.951878 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:31.952330 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:31.952439 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:31.953614 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:31.953622 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:31.953637 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:31.954288 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:31.954336 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:31.954356 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:31.954440 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:31.954618 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 02:55:31.954696 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:55:31.955971 23331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:32.013831 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:32.013272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45991 (local address 127.0.0.1:52228)
0925 02:55:32.013530 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:32.013648 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 02:55:32.015126 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:32.014627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45991 (local address 127.0.0.1:52236)
0925 02:55:32.014878 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:32.014967 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 02:55:32.080487 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 23209
I20260925 02:55:32.101794 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 22858
I20260925 02:55:32.103088 22919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:32.211934 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:32.211500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.65:45991 (local address 127.0.0.1:52244)
0925 02:55:32.211705 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:32.211794 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.65:45991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 02:55:32.212523 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 22858
2026-09-25T02:55:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2888 ms)
[----------] 6 tests from EncryptionPolicyTest (23025 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:32 dist-test-slave-g1xh krb5kdc[23388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:32 dist-test-slave-g1xh krb5kdc[23388](info): set up 2 sockets
Sep 25 02:55:32 dist-test-slave-g1xh krb5kdc[23388](info): commencing operation
krb5kdc: starting...
W20260925 02:55:33.350257 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:33 dist-test-slave-g1xh krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:55:33.643769 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:33.973378 23401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:33.973729 23401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:33.973906 23401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:33.981247 23401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:33.981408 23401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:33.981513 23401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:33.981633 23401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:33.981722 23401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:33.991751 23401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:33.993156 23401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:33.994449 23401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:34.006091 23406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:34.006104 23409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:34.006630 23407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:34.006999 23401 server_base.cc:1034] running on GCE node
I20260925 02:55:34.007804 23401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:34.009140 23401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:34.010361 23401 hybrid_clock.cc:648] HybridClock initialized: now 1790304934010317 us; error 37 us; skew 500 ppm
I20260925 02:55:34.010725 23401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:34 dist-test-slave-g1xh krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:34.013765 23401 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:55:34.015129 23401 webserver.cc:460] Webserver started at http://127.0.0.1:37635/ using document root <none> and password file <none>
I20260925 02:55:34.015897 23401 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:34.016044 23401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:34.016348 23401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:34.019249 23401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "ed95ac3e4fbc424cbe06a8360514267a"
format_stamp: "Formatted at 2026-09-25 02:55:34 on dist-test-slave-g1xh"
server_key: "01c82914c54130f1f0067d6f623c0da7"
server_key_iv: "13c2100fc18e9634ab3ddfce915b781a"
server_key_version: "encryptionkey@0"
I20260925 02:55:34.020020 23401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "ed95ac3e4fbc424cbe06a8360514267a"
format_stamp: "Formatted at 2026-09-25 02:55:34 on dist-test-slave-g1xh"
server_key: "01c82914c54130f1f0067d6f623c0da7"
server_key_iv: "13c2100fc18e9634ab3ddfce915b781a"
server_key_version: "encryptionkey@0"
I20260925 02:55:34.025233 23401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:55:34.028887 23416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:34.030346 23401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:34.030576 23401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "ed95ac3e4fbc424cbe06a8360514267a"
format_stamp: "Formatted at 2026-09-25 02:55:34 on dist-test-slave-g1xh"
server_key: "01c82914c54130f1f0067d6f623c0da7"
server_key_iv: "13c2100fc18e9634ab3ddfce915b781a"
server_key_version: "encryptionkey@0"
I20260925 02:55:34.030810 23401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:34.061380 23401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:34.065591 23401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:34.065905 23401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:34.084759 23401 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36763
I20260925 02:55:34.084786 23467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36763 every 8 connection(s)
I20260925 02:55:34.086161 23401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:34.089871 23468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:34.092170 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 23401
I20260925 02:55:34.092408 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:34.093263 2737 external_mini_cluster.cc:1428] Setting key 2be2033eef6b1adbda2c57454816278d
I20260925 02:55:34.102227 23468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a: Bootstrap starting.
I20260925 02:55:34.105463 23468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:34.106459 23468 log.cc:824] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:55:34 dist-test-slave-g1xh krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:55:34.109694 23468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a: No bootstrap required, opened a new log
I20260925 02:55:34.116437 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:34.095335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52085 (local address 127.0.0.1:36763)
0925 02:55:34.096215 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:34.096230 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:34.096278 (+ 48us) server_negotiation.cc:407] Connection header received
0925 02:55:34.097993 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:34.098017 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:34.098401 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:34.098818 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:34.100353 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:34.101623 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:34.102963 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:34.103707 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:34.103930 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:34.108620 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:34.108628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:34.108642 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:34.108679 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:34.111965 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:34.112909 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:34.112916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:34.112923 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:34.113088 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:34.113605 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:34.113609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:34.113611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:34.114148 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:34.114404 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:34.114934 (+ 530us) server_negotiation.cc:299] Negotiation successful
0925 02:55:34.115261 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":414,"threads_started":1}
I20260925 02:55:34.119168 23468 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed95ac3e4fbc424cbe06a8360514267a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36763 } }
I20260925 02:55:34.119711 23468 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:34.119841 23468 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed95ac3e4fbc424cbe06a8360514267a, State: Initialized, Role: FOLLOWER
I20260925 02:55:34.120452 23468 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed95ac3e4fbc424cbe06a8360514267a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36763 } }
I20260925 02:55:34.120777 23468 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:34.120924 23468 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:34.121003 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:34.121073 23468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:34.123246 23468 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed95ac3e4fbc424cbe06a8360514267a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36763 } }
I20260925 02:55:34.123812 23468 leader_election.cc:302] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed95ac3e4fbc424cbe06a8360514267a; no voters:
I20260925 02:55:34.124528 23468 leader_election.cc:288] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:34.124646 23476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:34.125815 23476 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [term 1 LEADER]: Becoming Leader. State: Replica: ed95ac3e4fbc424cbe06a8360514267a, State: Running, Role: LEADER
I20260925 02:55:34.126494 23476 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed95ac3e4fbc424cbe06a8360514267a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36763 } }
I20260925 02:55:34.127128 23468 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:34.131258 23479 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed95ac3e4fbc424cbe06a8360514267a. Latest consensus state: current_term: 1 leader_uuid: "ed95ac3e4fbc424cbe06a8360514267a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed95ac3e4fbc424cbe06a8360514267a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36763 } } }
I20260925 02:55:34.132491 23479 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:34.131791 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed95ac3e4fbc424cbe06a8360514267a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed95ac3e4fbc424cbe06a8360514267a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36763 } } }
I20260925 02:55:34.133123 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:34.140592 23485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:34.146427 23485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:34.147154 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:34.126489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54925 (local address 127.0.0.1:36763)
0925 02:55:34.126738 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:34.126747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:34.127632 (+ 885us) server_negotiation.cc:407] Connection header received
0925 02:55:34.127869 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:34.127874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:34.127954 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:34.128070 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:34.129645 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:34.130845 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:34.132824 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:34.133766 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:34.133917 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:34.138073 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:34.138082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:34.138090 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:34.138141 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:34.142111 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:34.143420 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:34.143430 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:34.143436 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:34.143661 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:34.144420 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:34.144429 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:34.144435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:34.145008 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:34.145326 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:34.146340 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0925 02:55:34.146606 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:55:34.158854 23485 catalog_manager.cc:1347] Generated new cluster ID: cc6e0351f87a422b9cb69437732b3a62
I20260925 02:55:34.159000 23485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:34.197045 23485 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:34.198973 23485 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:34.222471 23485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed95ac3e4fbc424cbe06a8360514267a: Generated new TSK 0
I20260925 02:55:34.223616 23485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:34.238502 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 23401
I20260925 02:55:34.239992 23463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:34.335994 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 23401
W20260925 02:55:34.354276 23386 connection.cc:535] client connection to 127.0.0.1:36763 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36763): Connection reset by peer (error 104)
2026-09-25T02:55:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2112 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:34 dist-test-slave-g1xh krb5kdc[23501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:34 dist-test-slave-g1xh krb5kdc[23501](info): set up 2 sockets
Sep 25 02:55:34 dist-test-slave-g1xh krb5kdc[23501](info): commencing operation
krb5kdc: starting...
W20260925 02:55:35.493983 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:35 dist-test-slave-g1xh krb5kdc[23501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:55:35.773313 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:36.107281 23514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:36.107667 23514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:36.107834 23514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:36.115078 23514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:36.115236 23514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:36.115350 23514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:36.115460 23514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:36.115540 23514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:36.125552 23514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:36.126896 23514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:36.128270 23514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:36.140537 23519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:36.140482 23520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:36.140515 23522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:36.141443 23514 server_base.cc:1034] running on GCE node
I20260925 02:55:36.142287 23514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:36.143839 23514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:36.145061 23514 hybrid_clock.cc:648] HybridClock initialized: now 1790304936144993 us; error 31 us; skew 500 ppm
I20260925 02:55:36.145516 23514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:36 dist-test-slave-g1xh krb5kdc[23501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:36.156163 23514 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:55:36.157872 23514 webserver.cc:460] Webserver started at http://127.0.0.1:36059/ using document root <none> and password file <none>
I20260925 02:55:36.158645 23514 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:36.158805 23514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:36.159118 23514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:36.162257 23514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "70aa8b97ca44457a9e810229e9dedb60"
format_stamp: "Formatted at 2026-09-25 02:55:36 on dist-test-slave-g1xh"
server_key: "233a86c88b7ba3e95378a0e2610ede72"
server_key_iv: "02bfb2c212d1922c6ea91eaee627f07c"
server_key_version: "encryptionkey@0"
I20260925 02:55:36.163014 23514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "70aa8b97ca44457a9e810229e9dedb60"
format_stamp: "Formatted at 2026-09-25 02:55:36 on dist-test-slave-g1xh"
server_key: "233a86c88b7ba3e95378a0e2610ede72"
server_key_iv: "02bfb2c212d1922c6ea91eaee627f07c"
server_key_version: "encryptionkey@0"
I20260925 02:55:36.168346 23514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:55:36.172034 23529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:36.173317 23514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 02:55:36.173506 23514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "70aa8b97ca44457a9e810229e9dedb60"
format_stamp: "Formatted at 2026-09-25 02:55:36 on dist-test-slave-g1xh"
server_key: "233a86c88b7ba3e95378a0e2610ede72"
server_key_iv: "02bfb2c212d1922c6ea91eaee627f07c"
server_key_version: "encryptionkey@0"
I20260925 02:55:36.173707 23514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:36.204075 23514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:36.208218 23514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:36.208540 23514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:36.227909 23514 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44077
I20260925 02:55:36.227954 23580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44077 every 8 connection(s)
I20260925 02:55:36.229357 23514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:36.231148 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 23514
I20260925 02:55:36.231333 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:36.231948 2737 external_mini_cluster.cc:1428] Setting key 0910ace2a15189c379528ac84b24f458
I20260925 02:55:36.234047 23581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:36.245663 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:36.233747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47471 (local address 127.0.0.1:44077)
0925 02:55:36.234567 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:36.234580 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:36.234620 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:55:36.236328 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:36.236348 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:36.236722 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:36.237046 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:36.237710 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:36.239080 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:36.241017 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:36.241664 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:36.241805 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:36.243183 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:36.243190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:36.243204 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:36.243860 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:36.243946 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:36.243974 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:36.244105 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:36.244425 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 02:55:36.244640 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":381,"threads_started":1}
I20260925 02:55:36.247076 23581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60: Bootstrap starting.
I20260925 02:55:36.249876 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:36.250815 23581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:36.252136 23581 log.cc:824] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:36.255117 23581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60: No bootstrap required, opened a new log
I20260925 02:55:36.262578 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:36.255301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60627 (local address 127.0.0.1:44077)
0925 02:55:36.255493 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:36.255498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:36.256043 (+ 545us) server_negotiation.cc:407] Connection header received
0925 02:55:36.256234 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:36.256241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:36.256328 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:36.256453 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:36.256971 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:36.258193 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:36.259597 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:36.260344 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:36.260492 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:36.261741 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:36.261748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:36.261753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:36.261775 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:36.261820 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:36.261840 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:36.261957 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:36.262159 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 02:55:36.262218 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:55:36.265285 23581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa8b97ca44457a9e810229e9dedb60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44077 } }
I20260925 02:55:36.265666 23581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:36.265776 23581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70aa8b97ca44457a9e810229e9dedb60, State: Initialized, Role: FOLLOWER
I20260925 02:55:36.266359 23581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa8b97ca44457a9e810229e9dedb60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44077 } }
I20260925 02:55:36.266598 23581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:36.266767 23581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:36.266956 23581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:36.269703 23581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa8b97ca44457a9e810229e9dedb60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44077 } }
I20260925 02:55:36.270293 23581 leader_election.cc:302] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70aa8b97ca44457a9e810229e9dedb60; no voters:
I20260925 02:55:36.270885 23581 leader_election.cc:288] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:36.271059 23591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:36.271781 23591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [term 1 LEADER]: Becoming Leader. State: Replica: 70aa8b97ca44457a9e810229e9dedb60, State: Running, Role: LEADER
I20260925 02:55:36.272336 23591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa8b97ca44457a9e810229e9dedb60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44077 } }
I20260925 02:55:36.273221 23581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:36.275341 23593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70aa8b97ca44457a9e810229e9dedb60. Latest consensus state: current_term: 1 leader_uuid: "70aa8b97ca44457a9e810229e9dedb60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa8b97ca44457a9e810229e9dedb60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44077 } } }
I20260925 02:55:36.275267 23592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70aa8b97ca44457a9e810229e9dedb60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa8b97ca44457a9e810229e9dedb60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44077 } } }
I20260925 02:55:36.275800 23593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:36.275837 23592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:36.277974 23598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:36.283524 23598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:36.292874 23598 catalog_manager.cc:1347] Generated new cluster ID: 580572c960df4eb38b59a092fe587a29
I20260925 02:55:36.293015 23598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:36.308243 23598 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:36.310151 23598 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:36.318337 23598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70aa8b97ca44457a9e810229e9dedb60: Generated new TSK 0
I20260925 02:55:36.319243 23598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:36.325403 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 23514
W20260925 02:55:36.327235 23576 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47471
W20260925 02:55:36.328089 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 23514: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:55:36.328295 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 23514
W20260925 02:55:36.346609 23499 connection.cc:535] client connection to 127.0.0.1:44077 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44077): Connection reset by peer (error 104)
2026-09-25T02:55:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:36 dist-test-slave-g1xh krb5kdc[23612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:36 dist-test-slave-g1xh krb5kdc[23612](info): set up 2 sockets
Sep 25 02:55:36 dist-test-slave-g1xh krb5kdc[23612](info): commencing operation
krb5kdc: starting...
W20260925 02:55:37.454545 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:37 dist-test-slave-g1xh krb5kdc[23612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:55:37.738765 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:38.066093 23625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:38.066385 23625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:38.066550 23625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:38.073858 23625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:38.074002 23625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:38.074108 23625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:38.074213 23625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:38.074299 23625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:38.084524 23625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:38.085844 23625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:38.087147 23625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:38.099345 23631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:38.099632 23630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:38.099529 23633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:38.100613 23625 server_base.cc:1034] running on GCE node
I20260925 02:55:38.101483 23625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:38.102922 23625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:38.104209 23625 hybrid_clock.cc:648] HybridClock initialized: now 1790304938104158 us; error 55 us; skew 500 ppm
I20260925 02:55:38.104624 23625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:38 dist-test-slave-g1xh krb5kdc[23612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:38.107976 23625 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:55:38.109563 23625 webserver.cc:460] Webserver started at http://127.0.0.1:45633/ using document root <none> and password file <none>
I20260925 02:55:38.110280 23625 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:38.110433 23625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:38.110755 23625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:38.113816 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "28223b5e06f5462abf9c653534190ba9"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-g1xh"
server_key: "61e42be3559900fa97ec69f72f24d389"
server_key_iv: "984e284a2a8f5caf1af5466a007ffb31"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.114590 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "28223b5e06f5462abf9c653534190ba9"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-g1xh"
server_key: "61e42be3559900fa97ec69f72f24d389"
server_key_iv: "984e284a2a8f5caf1af5466a007ffb31"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.120064 23625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 02:55:38.123737 23640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:38.125119 23625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:38.125314 23625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "28223b5e06f5462abf9c653534190ba9"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-g1xh"
server_key: "61e42be3559900fa97ec69f72f24d389"
server_key_iv: "984e284a2a8f5caf1af5466a007ffb31"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.125527 23625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:38.131673 23625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:38.135866 23625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:38.136121 23625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:38.154707 23625 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35561
I20260925 02:55:38.154932 23691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35561 every 8 connection(s)
I20260925 02:55:38.156282 23625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:38.160013 23692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:38.165454 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 23625
I20260925 02:55:38.165630 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:38.166117 2737 external_mini_cluster.cc:1428] Setting key 4bce01c97fb32ad0bdc643dd050ef9a3
I20260925 02:55:38.171587 23692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9: Bootstrap starting.
I20260925 02:55:38.174095 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:38.168077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51767 (local address 127.0.0.1:35561)
0925 02:55:38.168900 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:38.168912 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:38.168955 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:55:38.170670 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:38.170693 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:38.171039 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:38.171431 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:38.172510 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:38.172516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:38.172528 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:38.172542 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:38.172598 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:38.172618 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:38.172855 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:38.172988 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 02:55:38.173172 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":359,"threads_started":1}
I20260925 02:55:38.175056 23692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:38.176293 23692 log.cc:824] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:38.177790 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:38.179270 23692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9: No bootstrap required, opened a new log
I20260925 02:55:38.184775 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:38.182144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41757 (local address 127.0.0.1:35561)
0925 02:55:38.182314 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:38.182318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:38.182739 (+ 421us) server_negotiation.cc:407] Connection header received
0925 02:55:38.182937 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:38.182942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:38.183003 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:38.183110 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:38.184056 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:38.184066 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:38.184070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:38.184089 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:38.184139 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:38.184156 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:38.184298 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:38.184422 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 02:55:38.184493 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:55:38.188529 23692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28223b5e06f5462abf9c653534190ba9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35561 } }
I20260925 02:55:38.188949 23692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:38.189070 23692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28223b5e06f5462abf9c653534190ba9, State: Initialized, Role: FOLLOWER
I20260925 02:55:38.189704 23692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28223b5e06f5462abf9c653534190ba9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35561 } }
I20260925 02:55:38.189976 23692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:38.190122 23692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:38.190313 23692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:38.192822 23692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28223b5e06f5462abf9c653534190ba9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35561 } }
I20260925 02:55:38.193342 23692 leader_election.cc:302] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28223b5e06f5462abf9c653534190ba9; no voters:
I20260925 02:55:38.193954 23692 leader_election.cc:288] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:38.194084 23702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:38.194766 23702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [term 1 LEADER]: Becoming Leader. State: Replica: 28223b5e06f5462abf9c653534190ba9, State: Running, Role: LEADER
I20260925 02:55:38.195264 23702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28223b5e06f5462abf9c653534190ba9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35561 } }
I20260925 02:55:38.196095 23692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:38.198122 23704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28223b5e06f5462abf9c653534190ba9. Latest consensus state: current_term: 1 leader_uuid: "28223b5e06f5462abf9c653534190ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28223b5e06f5462abf9c653534190ba9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35561 } } }
I20260925 02:55:38.198268 23703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28223b5e06f5462abf9c653534190ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28223b5e06f5462abf9c653534190ba9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35561 } } }
I20260925 02:55:38.199062 23704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:38.199062 23703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:38.201341 23710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:38.205847 23710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:38.214334 23710 catalog_manager.cc:1347] Generated new cluster ID: 057a3a1f13e14d38a26160b92416798e
I20260925 02:55:38.214471 23710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:38.239004 23710 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:38.239531 23710 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:38.245819 23710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28223b5e06f5462abf9c653534190ba9: Generated new TSK 0
I20260925 02:55:38.246807 23710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:38.256846 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 23625
W20260925 02:55:38.257967 23687 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51767
W20260925 02:55:38.258699 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 23625: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:55:38.258846 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 23625
2026-09-25T02:55:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:38 dist-test-slave-g1xh krb5kdc[23723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:38 dist-test-slave-g1xh krb5kdc[23723](info): set up 2 sockets
Sep 25 02:55:38 dist-test-slave-g1xh krb5kdc[23723](info): commencing operation
krb5kdc: starting...
W20260925 02:55:39.367053 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:39 dist-test-slave-g1xh krb5kdc[23723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:55:39.644901 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:39.973523 23736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:39.973872 23736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:39.974047 23736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:39.981140 23736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:39.981302 23736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:39.981417 23736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:39.981528 23736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:39.981618 23736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:39.991360 23736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:39.992708 23736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:39.994012 23736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:40.005954 23742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:40.005963 23744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:40.006014 23741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:40.007076 23736 server_base.cc:1034] running on GCE node
I20260925 02:55:40.007920 23736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:40.009356 23736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:40.010582 23736 hybrid_clock.cc:648] HybridClock initialized: now 1790304940010545 us; error 49 us; skew 500 ppm
I20260925 02:55:40.010931 23736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:40 dist-test-slave-g1xh krb5kdc[23723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:40.013995 23736 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:55:40.015502 23736 webserver.cc:460] Webserver started at http://127.0.0.1:32833/ using document root <none> and password file <none>
I20260925 02:55:40.016232 23736 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:40.016383 23736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:40.016681 23736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:40.019762 23736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "3ed8e87f09de472d9022ef08d839de81"
format_stamp: "Formatted at 2026-09-25 02:55:40 on dist-test-slave-g1xh"
server_key: "37e1bfab3ff28f003241c7149b73cd55"
server_key_iv: "d9568f6d1ea6763f07320e4b1b06b7e1"
server_key_version: "encryptionkey@0"
I20260925 02:55:40.020516 23736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "3ed8e87f09de472d9022ef08d839de81"
format_stamp: "Formatted at 2026-09-25 02:55:40 on dist-test-slave-g1xh"
server_key: "37e1bfab3ff28f003241c7149b73cd55"
server_key_iv: "d9568f6d1ea6763f07320e4b1b06b7e1"
server_key_version: "encryptionkey@0"
I20260925 02:55:40.025696 23736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 02:55:40.029276 23751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:40.030651 23736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:55:40.030871 23736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "3ed8e87f09de472d9022ef08d839de81"
format_stamp: "Formatted at 2026-09-25 02:55:40 on dist-test-slave-g1xh"
server_key: "37e1bfab3ff28f003241c7149b73cd55"
server_key_iv: "d9568f6d1ea6763f07320e4b1b06b7e1"
server_key_version: "encryptionkey@0"
I20260925 02:55:40.031097 23736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:40.048998 23736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:40.052898 23736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:40.053208 23736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:40.072069 23736 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37325
I20260925 02:55:40.072109 23802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37325 every 8 connection(s)
I20260925 02:55:40.073419 23736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:40.077189 23803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:40.081153 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 23736
I20260925 02:55:40.081350 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:40.081799 2737 external_mini_cluster.cc:1428] Setting key 1dcb958115d8a52a186bed3eb159e77f
I20260925 02:55:40.090592 23803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81: Bootstrap starting.
I20260925 02:55:40.094480 23803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:40.095624 23803 log.cc:824] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:55:40 dist-test-slave-g1xh krb5kdc[23723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:55:40.098497 23803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81: No bootstrap required, opened a new log
I20260925 02:55:40.105361 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:40.083610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40703 (local address 127.0.0.1:37325)
0925 02:55:40.084384 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:40.084399 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:40.084451 (+ 52us) server_negotiation.cc:407] Connection header received
0925 02:55:40.086347 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:40.086381 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:40.086909 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:40.087352 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:40.088657 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:40.090677 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:40.092010 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:40.093127 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:40.093316 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:40.097647 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:40.097659 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:40.097675 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:40.097717 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:40.101044 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:40.101836 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:40.101845 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:40.101854 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:40.102025 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:40.102454 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:40.102458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:40.102460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:40.103054 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:40.103410 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:40.103859 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 02:55:40.104303 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":320,"threads_started":1}
I20260925 02:55:40.106468 23803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed8e87f09de472d9022ef08d839de81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260925 02:55:40.106954 23803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:40.107069 23803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed8e87f09de472d9022ef08d839de81, State: Initialized, Role: FOLLOWER
I20260925 02:55:40.107746 23803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed8e87f09de472d9022ef08d839de81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260925 02:55:40.108083 23803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:40.108229 23803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:40.108422 23803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:40.109395 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:40.110911 23803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed8e87f09de472d9022ef08d839de81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260925 02:55:40.111410 23803 leader_election.cc:302] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ed8e87f09de472d9022ef08d839de81; no voters:
I20260925 02:55:40.112079 23803 leader_election.cc:288] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:40.112315 23808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:40.113711 23808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [term 1 LEADER]: Becoming Leader. State: Replica: 3ed8e87f09de472d9022ef08d839de81, State: Running, Role: LEADER
I20260925 02:55:40.114544 23803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:40.114511 23808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed8e87f09de472d9022ef08d839de81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260925 02:55:40.120303 23810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ed8e87f09de472d9022ef08d839de81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed8e87f09de472d9022ef08d839de81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } } }
I20260925 02:55:40.120728 23810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:40.121263 23812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ed8e87f09de472d9022ef08d839de81. Latest consensus state: current_term: 1 leader_uuid: "3ed8e87f09de472d9022ef08d839de81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed8e87f09de472d9022ef08d839de81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } } }
I20260925 02:55:40.121609 23812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:40.130043 23818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:40.136163 23818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:40.149582 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:40.115803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40207 (local address 127.0.0.1:37325)
0925 02:55:40.115992 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:40.116001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:40.130269 (+ 14268us) server_negotiation.cc:407] Connection header received
0925 02:55:40.130524 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:40.130533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:40.130649 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:40.130800 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:40.132942 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:40.134083 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:40.136334 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:40.137114 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:40.137231 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:40.140773 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:40.140780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:40.140784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:40.140831 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:40.144534 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:40.145461 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:40.145467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:40.145470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:40.145624 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:40.146205 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:40.146214 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:40.146220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:40.146800 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:40.147101 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:40.148700 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0925 02:55:40.149005 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:55:40.153090 23818 catalog_manager.cc:1347] Generated new cluster ID: 96f135d032c749abb33876a96471da69
I20260925 02:55:40.153215 23818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:40.186167 23818 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:40.187599 23818 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:40.193053 23818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ed8e87f09de472d9022ef08d839de81: Generated new TSK 0
I20260925 02:55:40.193876 23818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:40.206671 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 23736
I20260925 02:55:40.207988 23798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:40.302220 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 23736
2026-09-25T02:55:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2044 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:40 dist-test-slave-g1xh krb5kdc[23836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:40 dist-test-slave-g1xh krb5kdc[23836](info): set up 2 sockets
Sep 25 02:55:40 dist-test-slave-g1xh krb5kdc[23836](info): commencing operation
krb5kdc: starting...
W20260925 02:55:41.426065 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.006s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:41 dist-test-slave-g1xh krb5kdc[23836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:55:41.719851 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:42.049434 23849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:42.049746 23849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:42.049841 23849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:42.057179 23849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:42.057298 23849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:42.057345 23849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:42.057408 23849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:42.057453 23849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:42.067494 23849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:42.068832 23849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:42.070120 23849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:42.082455 23854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:42.082484 23855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:42.083190 23857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:42.083338 23849 server_base.cc:1034] running on GCE node
I20260925 02:55:42.084309 23849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:42.085770 23849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:42.087030 23849 hybrid_clock.cc:648] HybridClock initialized: now 1790304942086975 us; error 49 us; skew 500 ppm
I20260925 02:55:42.087482 23849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:42 dist-test-slave-g1xh krb5kdc[23836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:42.096647 23849 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:55:42.098351 23849 webserver.cc:460] Webserver started at http://127.0.0.1:37685/ using document root <none> and password file <none>
I20260925 02:55:42.099139 23849 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:42.099308 23849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:42.099706 23849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:42.102792 23849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "69c51b3db41a49a08deb3e3d51c71021"
format_stamp: "Formatted at 2026-09-25 02:55:42 on dist-test-slave-g1xh"
server_key: "4a06d5185a3a92f7a5f0eb17268f9e23"
server_key_iv: "01de16587edc392fbe071c361cb7afbd"
server_key_version: "encryptionkey@0"
I20260925 02:55:42.103675 23849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "69c51b3db41a49a08deb3e3d51c71021"
format_stamp: "Formatted at 2026-09-25 02:55:42 on dist-test-slave-g1xh"
server_key: "4a06d5185a3a92f7a5f0eb17268f9e23"
server_key_iv: "01de16587edc392fbe071c361cb7afbd"
server_key_version: "encryptionkey@0"
I20260925 02:55:42.109073 23849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 02:55:42.112848 23864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:42.114140 23849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 02:55:42.114343 23849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "69c51b3db41a49a08deb3e3d51c71021"
format_stamp: "Formatted at 2026-09-25 02:55:42 on dist-test-slave-g1xh"
server_key: "4a06d5185a3a92f7a5f0eb17268f9e23"
server_key_iv: "01de16587edc392fbe071c361cb7afbd"
server_key_version: "encryptionkey@0"
I20260925 02:55:42.114560 23849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:42.141131 23849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:42.145684 23849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:42.146003 23849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:42.165120 23849 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34341
I20260925 02:55:42.165144 23915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34341 every 8 connection(s)
I20260925 02:55:42.166546 23849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:42.167683 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 23849
I20260925 02:55:42.167904 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:42.168362 2737 external_mini_cluster.cc:1428] Setting key 602cff327010b8dd8fdac13d0ca5b409
I20260925 02:55:42.171025 23916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:42.183907 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:42.170249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33725 (local address 127.0.0.1:34341)
0925 02:55:42.171132 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:42.171149 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:42.171199 (+ 50us) server_negotiation.cc:407] Connection header received
0925 02:55:42.173593 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:42.173621 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:42.174161 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:42.174619 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:42.175153 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:42.177219 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:42.178564 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:42.179630 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:42.179838 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:42.180996 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:42.181011 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:42.181030 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:42.181825 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:42.181921 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:42.181957 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:42.182069 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:42.182265 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 02:55:42.182551 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":346,"threads_started":1}
I20260925 02:55:42.185141 23916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021: Bootstrap starting.
I20260925 02:55:42.187598 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:42.188607 23916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:42.189857 23916 log.cc:824] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:42.193055 23916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021: No bootstrap required, opened a new log
I20260925 02:55:42.200107 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:42.193108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50089 (local address 127.0.0.1:34341)
0925 02:55:42.193330 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:42.193336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:42.193723 (+ 387us) server_negotiation.cc:407] Connection header received
0925 02:55:42.193923 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:42.193926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:42.193996 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:42.194077 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:42.194682 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:42.195473 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:42.197460 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:42.197931 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:42.198021 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:42.199287 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:42.199292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:42.199295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:42.199317 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:42.199373 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:42.199389 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:42.199463 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:42.199649 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 02:55:42.199759 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:55:42.203316 23916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c51b3db41a49a08deb3e3d51c71021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34341 } }
I20260925 02:55:42.203824 23916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:42.203936 23916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c51b3db41a49a08deb3e3d51c71021, State: Initialized, Role: FOLLOWER
I20260925 02:55:42.204538 23916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c51b3db41a49a08deb3e3d51c71021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34341 } }
I20260925 02:55:42.204782 23916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:42.204957 23916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:42.205147 23916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:42.208055 23916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c51b3db41a49a08deb3e3d51c71021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34341 } }
I20260925 02:55:42.208631 23916 leader_election.cc:302] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c51b3db41a49a08deb3e3d51c71021; no voters:
I20260925 02:55:42.209262 23916 leader_election.cc:288] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:42.209425 23926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:42.210163 23926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [term 1 LEADER]: Becoming Leader. State: Replica: 69c51b3db41a49a08deb3e3d51c71021, State: Running, Role: LEADER
I20260925 02:55:42.210811 23926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c51b3db41a49a08deb3e3d51c71021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34341 } }
I20260925 02:55:42.211619 23916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:42.213739 23928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c51b3db41a49a08deb3e3d51c71021. Latest consensus state: current_term: 1 leader_uuid: "69c51b3db41a49a08deb3e3d51c71021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c51b3db41a49a08deb3e3d51c71021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34341 } } }
I20260925 02:55:42.214440 23928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:42.213915 23927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69c51b3db41a49a08deb3e3d51c71021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c51b3db41a49a08deb3e3d51c71021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34341 } } }
I20260925 02:55:42.215047 23927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:42.215636 23932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:42.222014 23932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:42.230778 23932 catalog_manager.cc:1347] Generated new cluster ID: 39778d0aefc04562a77d28cade9d9598
I20260925 02:55:42.230916 23932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:42.244801 23932 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:42.246409 23932 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:42.259933 23932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69c51b3db41a49a08deb3e3d51c71021: Generated new TSK 0
I20260925 02:55:42.260810 23932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:42.269016 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 23849
W20260925 02:55:42.270203 23911 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33725
W20260925 02:55:42.270953 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 23849: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:55:42.271133 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 23849
2026-09-25T02:55:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:42 dist-test-slave-g1xh krb5kdc[23947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:42 dist-test-slave-g1xh krb5kdc[23947](info): set up 2 sockets
Sep 25 02:55:42 dist-test-slave-g1xh krb5kdc[23947](info): commencing operation
krb5kdc: starting...
W20260925 02:55:43.390090 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:43 dist-test-slave-g1xh krb5kdc[23947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:55:43.683724 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:44.027310 23960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:44.027707 23960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:44.027876 23960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:44.035270 23960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:44.035392 23960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:44.035513 23960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:44.035641 23960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:44.035696 23960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:44.045753 23960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:44.047109 23960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:44.048482 23960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:44.060209 23966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:44.060228 23968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:44.061029 23965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:44.061009 23960 server_base.cc:1034] running on GCE node
I20260925 02:55:44.061968 23960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:44.063414 23960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:44.064642 23960 hybrid_clock.cc:648] HybridClock initialized: now 1790304944064607 us; error 52 us; skew 500 ppm
I20260925 02:55:44.065021 23960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:44 dist-test-slave-g1xh krb5kdc[23947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:44.068194 23960 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:55:44.069772 23960 webserver.cc:460] Webserver started at http://127.0.0.1:42017/ using document root <none> and password file <none>
I20260925 02:55:44.070478 23960 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:44.070628 23960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:44.070948 23960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:44.074009 23960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "1d5018c16cad40ea86a98598ff78138d"
format_stamp: "Formatted at 2026-09-25 02:55:44 on dist-test-slave-g1xh"
server_key: "5f7d983acd75f1901348a9130605beee"
server_key_iv: "69174326e4a10c80addbb0dc06be4541"
server_key_version: "encryptionkey@0"
I20260925 02:55:44.074826 23960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "1d5018c16cad40ea86a98598ff78138d"
format_stamp: "Formatted at 2026-09-25 02:55:44 on dist-test-slave-g1xh"
server_key: "5f7d983acd75f1901348a9130605beee"
server_key_iv: "69174326e4a10c80addbb0dc06be4541"
server_key_version: "encryptionkey@0"
I20260925 02:55:44.080184 23960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 02:55:44.083675 23975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:44.085037 23960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:44.085234 23960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "1d5018c16cad40ea86a98598ff78138d"
format_stamp: "Formatted at 2026-09-25 02:55:44 on dist-test-slave-g1xh"
server_key: "5f7d983acd75f1901348a9130605beee"
server_key_iv: "69174326e4a10c80addbb0dc06be4541"
server_key_version: "encryptionkey@0"
I20260925 02:55:44.085449 23960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:44.091323 23960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:44.095424 23960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:44.095780 23960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:44.114648 23960 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39613
I20260925 02:55:44.114689 24026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39613 every 8 connection(s)
I20260925 02:55:44.116086 23960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:44.119907 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:44.122967 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 23960
I20260925 02:55:44.123158 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:44.123703 2737 external_mini_cluster.cc:1428] Setting key 7557b210e75fdbba396283392c2f94c4
I20260925 02:55:44.131457 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:44.125559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55165 (local address 127.0.0.1:39613)
0925 02:55:44.126358 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:44.126369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:44.126411 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:55:44.128126 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:44.128149 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:44.128476 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:44.128870 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:44.129922 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:44.129929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:44.129944 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:44.129966 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:44.130016 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:44.130038 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:44.130280 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:44.130409 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 02:55:44.130586 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":361,"threads_started":1}
I20260925 02:55:44.132062 24027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d: Bootstrap starting.
I20260925 02:55:44.135031 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:44.135488 24027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:44.136543 24027 log.cc:824] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:44.139909 24027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d: No bootstrap required, opened a new log
I20260925 02:55:44.142263 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:44.139595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51805 (local address 127.0.0.1:39613)
0925 02:55:44.139789 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:44.139799 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:44.140269 (+ 470us) server_negotiation.cc:407] Connection header received
0925 02:55:44.140474 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:44.140478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:44.140523 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:44.140607 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:44.141575 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:44.141579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:44.141583 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:44.141597 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:44.141634 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:44.141648 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:44.141752 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:44.141872 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 02:55:44.141923 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:55:44.148610 24027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5018c16cad40ea86a98598ff78138d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39613 } }
I20260925 02:55:44.149020 24027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:44.149130 24027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5018c16cad40ea86a98598ff78138d, State: Initialized, Role: FOLLOWER
I20260925 02:55:44.149715 24027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5018c16cad40ea86a98598ff78138d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39613 } }
I20260925 02:55:44.150039 24027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:44.150158 24027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:44.150286 24027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:44.152451 24027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5018c16cad40ea86a98598ff78138d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39613 } }
I20260925 02:55:44.152897 24027 leader_election.cc:302] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d5018c16cad40ea86a98598ff78138d; no voters:
I20260925 02:55:44.153406 24027 leader_election.cc:288] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:44.153555 24037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:44.154280 24037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [term 1 LEADER]: Becoming Leader. State: Replica: 1d5018c16cad40ea86a98598ff78138d, State: Running, Role: LEADER
I20260925 02:55:44.154860 24037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5018c16cad40ea86a98598ff78138d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39613 } }
I20260925 02:55:44.155525 24027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:44.157866 24038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d5018c16cad40ea86a98598ff78138d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5018c16cad40ea86a98598ff78138d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39613 } } }
I20260925 02:55:44.158401 24038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:44.159181 24039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d5018c16cad40ea86a98598ff78138d. Latest consensus state: current_term: 1 leader_uuid: "1d5018c16cad40ea86a98598ff78138d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5018c16cad40ea86a98598ff78138d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39613 } } }
I20260925 02:55:44.159582 24039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:44.163112 24044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:44.167661 24044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:44.176107 24044 catalog_manager.cc:1347] Generated new cluster ID: 2676614086e345c4a65190e47c9ff5b9
I20260925 02:55:44.176239 24044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:44.194806 24044 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:44.195395 24044 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:44.208678 24044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d5018c16cad40ea86a98598ff78138d: Generated new TSK 0
I20260925 02:55:44.209800 24044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:44.224251 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 23960
W20260925 02:55:44.225561 24022 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55165
W20260925 02:55:44.226541 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 23960: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:55:44.226702 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 23960
2026-09-25T02:55:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:44 dist-test-slave-g1xh krb5kdc[24058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:44 dist-test-slave-g1xh krb5kdc[24058](info): set up 2 sockets
Sep 25 02:55:44 dist-test-slave-g1xh krb5kdc[24058](info): commencing operation
krb5kdc: starting...
W20260925 02:55:45.358016 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:45 dist-test-slave-g1xh krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:55:45.651322 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:45.985549 24071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:45.985924 24071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:45.986099 24071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:45.993511 24071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:45.993685 24071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:45.993796 24071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:45.993919 24071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:45.993999 24071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:46.004187 24071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:46.005612 24071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:46.006963 24071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:46.019693 24076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:46.019658 24079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:46.019610 24077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:46.020812 24071 server_base.cc:1034] running on GCE node
I20260925 02:55:46.021665 24071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:46.023193 24071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:46.024436 24071 hybrid_clock.cc:648] HybridClock initialized: now 1790304946024400 us; error 58 us; skew 500 ppm
I20260925 02:55:46.024823 24071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:46 dist-test-slave-g1xh krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:46.028232 24071 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:55:46.029783 24071 webserver.cc:460] Webserver started at http://127.0.0.1:42377/ using document root <none> and password file <none>
I20260925 02:55:46.030503 24071 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:46.030669 24071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:46.030989 24071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:46.034196 24071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "8a669cab59594ec9a90d02467b042428"
format_stamp: "Formatted at 2026-09-25 02:55:46 on dist-test-slave-g1xh"
server_key: "237494dd4be6276e36811bedb41db49d"
server_key_iv: "ebb4fec275e7d2e8cead0c3d206a7ee1"
server_key_version: "encryptionkey@0"
I20260925 02:55:46.035006 24071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "8a669cab59594ec9a90d02467b042428"
format_stamp: "Formatted at 2026-09-25 02:55:46 on dist-test-slave-g1xh"
server_key: "237494dd4be6276e36811bedb41db49d"
server_key_iv: "ebb4fec275e7d2e8cead0c3d206a7ee1"
server_key_version: "encryptionkey@0"
I20260925 02:55:46.040410 24071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 02:55:46.043802 24086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:46.045102 24071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:55:46.045296 24071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "8a669cab59594ec9a90d02467b042428"
format_stamp: "Formatted at 2026-09-25 02:55:46 on dist-test-slave-g1xh"
server_key: "237494dd4be6276e36811bedb41db49d"
server_key_iv: "ebb4fec275e7d2e8cead0c3d206a7ee1"
server_key_version: "encryptionkey@0"
I20260925 02:55:46.045503 24071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:46.067473 24071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:46.072113 24071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:46.072429 24071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:46.092948 24071 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41183
I20260925 02:55:46.092994 24137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41183 every 8 connection(s)
I20260925 02:55:46.094398 24071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:46.098351 24138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:46.099809 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 24071
I20260925 02:55:46.100031 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:46.100565 2737 external_mini_cluster.cc:1428] Setting key 095ebef761cc0d441cab31c79e379eb7
I20260925 02:55:46.113493 24138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428: Bootstrap starting.
Sep 25 02:55:46 dist-test-slave-g1xh krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:55:46.117478 24138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:46.118664 24138 log.cc:824] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:46.121963 24138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428: No bootstrap required, opened a new log
I20260925 02:55:46.125941 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:46.102683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:50145 (local address 127.0.0.1:41183)
0925 02:55:46.103700 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:46.103717 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:46.103769 (+ 52us) server_negotiation.cc:407] Connection header received
0925 02:55:46.105470 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:46.105498 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:46.105971 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:46.106311 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:46.108263 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:46.109611 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:46.111512 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:46.112323 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:46.112465 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:46.117445 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:46.117454 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:46.117472 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:46.117514 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:46.121372 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:46.122340 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:46.122347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:46.122354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:46.122531 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:46.123041 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:46.123051 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:46.123053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:46.123657 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:46.124030 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:46.124561 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 02:55:46.124897 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":460,"threads_started":1}
I20260925 02:55:46.129990 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:46.130564 24138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a669cab59594ec9a90d02467b042428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41183 } }
I20260925 02:55:46.130950 24138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:46.131057 24138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a669cab59594ec9a90d02467b042428, State: Initialized, Role: FOLLOWER
I20260925 02:55:46.131744 24138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a669cab59594ec9a90d02467b042428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41183 } }
I20260925 02:55:46.131996 24138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:46.132153 24138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:46.132333 24138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:46.134910 24138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a669cab59594ec9a90d02467b042428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41183 } }
I20260925 02:55:46.135710 24138 leader_election.cc:302] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a669cab59594ec9a90d02467b042428; no voters:
I20260925 02:55:46.137940 24138 leader_election.cc:288] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:46.138113 24148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:46.139294 24148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [term 1 LEADER]: Becoming Leader. State: Replica: 8a669cab59594ec9a90d02467b042428, State: Running, Role: LEADER
I20260925 02:55:46.139966 24148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a669cab59594ec9a90d02467b042428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41183 } }
I20260925 02:55:46.140367 24138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:46.142906 24149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a669cab59594ec9a90d02467b042428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a669cab59594ec9a90d02467b042428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41183 } } }
I20260925 02:55:46.143348 24149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:46.148702 24153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:46.149812 24150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a669cab59594ec9a90d02467b042428. Latest consensus state: current_term: 1 leader_uuid: "8a669cab59594ec9a90d02467b042428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a669cab59594ec9a90d02467b042428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41183 } } }
I20260925 02:55:46.150290 24150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:46.156073 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:46.134956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:36541 (local address 127.0.0.1:41183)
0925 02:55:46.135162 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:46.135170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:46.135738 (+ 568us) server_negotiation.cc:407] Connection header received
0925 02:55:46.135958 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:46.135966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:46.136039 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:46.136160 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:46.137668 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:46.139167 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:46.141068 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:46.141926 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:46.143018 (+ 1092us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:46.146437 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:46.146447 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:46.146455 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:46.146516 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:46.151432 (+ 4916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:46.152777 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:46.152782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:46.152785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:46.152941 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:46.153602 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:46.153606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:46.153610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:46.154046 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:46.155277 (+ 1231us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:46.155468 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 02:55:46.155689 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:55:46.156914 24153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:46.168483 24153 catalog_manager.cc:1347] Generated new cluster ID: 1b728e4c960449a58aba4c9cf934ca94
I20260925 02:55:46.168640 24153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:46.188241 24153 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:46.190250 24153 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:46.213117 24153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a669cab59594ec9a90d02467b042428: Generated new TSK 0
I20260925 02:55:46.214040 24153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:46.224611 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24071
I20260925 02:55:46.225869 24133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:46.318722 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 24071
W20260925 02:55:46.336759 24056 connection.cc:535] client connection to 127.0.0.1:41183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41183): Connection reset by peer (error 104)
2026-09-25T02:55:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:46 dist-test-slave-g1xh krb5kdc[24170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:46 dist-test-slave-g1xh krb5kdc[24170](info): set up 2 sockets
Sep 25 02:55:46 dist-test-slave-g1xh krb5kdc[24170](info): commencing operation
krb5kdc: starting...
W20260925 02:55:47.450147 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:47 dist-test-slave-g1xh krb5kdc[24170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:55:47.750686 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:48.084654 24183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:48.084957 24183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:48.085135 24183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:48.092478 24183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:48.092605 24183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:48.092700 24183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:48.092809 24183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:48.092882 24183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:48.103183 24183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:48.104617 24183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:48.106086 24183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:48.118127 24188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:48.118050 24191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:48.118085 24189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:48.118961 24183 server_base.cc:1034] running on GCE node
I20260925 02:55:48.119783 24183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:48.121177 24183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:48.122443 24183 hybrid_clock.cc:648] HybridClock initialized: now 1790304948122414 us; error 44 us; skew 500 ppm
I20260925 02:55:48.122788 24183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:48 dist-test-slave-g1xh krb5kdc[24170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:48.130307 24183 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:55:48.131875 24183 webserver.cc:460] Webserver started at http://127.0.0.1:46521/ using document root <none> and password file <none>
I20260925 02:55:48.132614 24183 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:48.132772 24183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:48.133080 24183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:48.136040 24183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "9a618592f7f94ffbaa42e1b69a2d2b39"
format_stamp: "Formatted at 2026-09-25 02:55:48 on dist-test-slave-g1xh"
server_key: "6d7bbfbf6ecc214d84d3f843fa5bfd46"
server_key_iv: "87dba8d122996d3e49b2d6267666ff6f"
server_key_version: "encryptionkey@0"
I20260925 02:55:48.136816 24183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "9a618592f7f94ffbaa42e1b69a2d2b39"
format_stamp: "Formatted at 2026-09-25 02:55:48 on dist-test-slave-g1xh"
server_key: "6d7bbfbf6ecc214d84d3f843fa5bfd46"
server_key_iv: "87dba8d122996d3e49b2d6267666ff6f"
server_key_version: "encryptionkey@0"
I20260925 02:55:48.141817 24183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:55:48.145177 24198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:48.146482 24183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:55:48.146664 24183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "9a618592f7f94ffbaa42e1b69a2d2b39"
format_stamp: "Formatted at 2026-09-25 02:55:48 on dist-test-slave-g1xh"
server_key: "6d7bbfbf6ecc214d84d3f843fa5bfd46"
server_key_iv: "87dba8d122996d3e49b2d6267666ff6f"
server_key_version: "encryptionkey@0"
I20260925 02:55:48.146863 24183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:48.172227 24183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:48.176414 24183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:48.176730 24183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:48.196015 24183 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42895
I20260925 02:55:48.196048 24249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42895 every 8 connection(s)
I20260925 02:55:48.197433 24183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:48.201339 24250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:48.207945 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 24183
I20260925 02:55:48.208154 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:48.208768 2737 external_mini_cluster.cc:1428] Setting key 4751959544e60b67aef9d269d071d76c
I20260925 02:55:48.213024 24250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39: Bootstrap starting.
I20260925 02:55:48.216444 24250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:48.217494 24250 log.cc:824] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:48.221091 24250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39: No bootstrap required, opened a new log
I20260925 02:55:48.222728 24253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:48.210837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:55665 (local address 127.0.0.1:42895)
0925 02:55:48.211672 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:48.211685 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:48.211724 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:55:48.213567 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:48.213590 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:48.214002 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:48.214389 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:48.214957 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:48.216311 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:48.217643 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:48.218736 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:48.218958 (+ 222us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:48.220438 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:48.220445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:48.220460 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:48.221022 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:48.221103 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:48.221127 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:48.221313 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:48.221536 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 02:55:48.221734 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":350,"threads_started":1}
I20260925 02:55:48.226600 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:48.231034 24250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a618592f7f94ffbaa42e1b69a2d2b39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42895 } }
I20260925 02:55:48.231606 24250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:48.231716 24250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a618592f7f94ffbaa42e1b69a2d2b39, State: Initialized, Role: FOLLOWER
I20260925 02:55:48.232435 24250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a618592f7f94ffbaa42e1b69a2d2b39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42895 } }
I20260925 02:55:48.232702 24250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:48.232861 24250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:48.233067 24250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:48.235477 24250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a618592f7f94ffbaa42e1b69a2d2b39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42895 } }
I20260925 02:55:48.236064 24250 leader_election.cc:302] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a618592f7f94ffbaa42e1b69a2d2b39; no voters:
I20260925 02:55:48.236801 24250 leader_election.cc:288] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:48.236927 24260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:48.237629 24260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [term 1 LEADER]: Becoming Leader. State: Replica: 9a618592f7f94ffbaa42e1b69a2d2b39, State: Running, Role: LEADER
I20260925 02:55:48.238286 24260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a618592f7f94ffbaa42e1b69a2d2b39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42895 } }
I20260925 02:55:48.239180 24253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:48.231955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:42225 (local address 127.0.0.1:42895)
0925 02:55:48.232145 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:48.232152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:48.232682 (+ 530us) server_negotiation.cc:407] Connection header received
0925 02:55:48.232980 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:48.232987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:48.233076 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:48.233174 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:48.233704 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:48.234489 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:48.235909 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:48.236448 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:48.236587 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:48.238155 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:48.238164 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:48.238169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:48.238197 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:48.238233 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:48.238248 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:48.238389 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:48.238798 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 02:55:48.238853 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:55:48.239281 24250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:48.241357 24261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a618592f7f94ffbaa42e1b69a2d2b39. Latest consensus state: current_term: 1 leader_uuid: "9a618592f7f94ffbaa42e1b69a2d2b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a618592f7f94ffbaa42e1b69a2d2b39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42895 } } }
I20260925 02:55:48.241973 24261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:48.241860 24262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a618592f7f94ffbaa42e1b69a2d2b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a618592f7f94ffbaa42e1b69a2d2b39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42895 } } }
I20260925 02:55:48.242835 24262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:48.244045 24266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:48.250001 24266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:48.259690 24266 catalog_manager.cc:1347] Generated new cluster ID: a8cb5bd0690443149f5c4ec7a6a6f2e8
I20260925 02:55:48.259840 24266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:48.277453 24266 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:48.279006 24266 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:48.287492 24266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a618592f7f94ffbaa42e1b69a2d2b39: Generated new TSK 0
I20260925 02:55:48.288481 24266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:48.293541 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24183
W20260925 02:55:48.294879 24245 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.142:55665
W20260925 02:55:48.295712 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24183: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:55:48.295869 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 24183
W20260925 02:55:48.313226 24168 connection.cc:535] client connection to 127.0.0.1:42895 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42895): Connection reset by peer (error 104)
2026-09-25T02:55:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:48 dist-test-slave-g1xh krb5kdc[24281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:48 dist-test-slave-g1xh krb5kdc[24281](info): set up 2 sockets
Sep 25 02:55:48 dist-test-slave-g1xh krb5kdc[24281](info): commencing operation
krb5kdc: starting...
W20260925 02:55:49.414011 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:49 dist-test-slave-g1xh krb5kdc[24281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:55:49.702893 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:50.036826 24294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:50.037114 24294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:50.037276 24294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:50.044730 24294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:50.044843 24294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:50.044958 24294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:50.045073 24294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:50.045141 24294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:50.055202 24294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:50.056496 24294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:50.057832 24294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:50.069497 24300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:50.069753 24299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:50.069990 24302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:50.070518 24294 server_base.cc:1034] running on GCE node
I20260925 02:55:50.071273 24294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:50.072710 24294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:50.073951 24294 hybrid_clock.cc:648] HybridClock initialized: now 1790304950073910 us; error 54 us; skew 500 ppm
I20260925 02:55:50.074303 24294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:50 dist-test-slave-g1xh krb5kdc[24281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:50.077527 24294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:55:50.079124 24294 webserver.cc:460] Webserver started at http://127.0.0.1:37641/ using document root <none> and password file <none>
I20260925 02:55:50.079916 24294 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:50.080062 24294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:50.080370 24294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:50.083473 24294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "fa0201da764748a7aad3b07a7ad31338"
format_stamp: "Formatted at 2026-09-25 02:55:50 on dist-test-slave-g1xh"
server_key: "6c05d47ca5d96067d05f60532d4647ef"
server_key_iv: "f38df8fd4a0883c5231fe4a67d7b50f9"
server_key_version: "encryptionkey@0"
I20260925 02:55:50.084342 24294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "fa0201da764748a7aad3b07a7ad31338"
format_stamp: "Formatted at 2026-09-25 02:55:50 on dist-test-slave-g1xh"
server_key: "6c05d47ca5d96067d05f60532d4647ef"
server_key_iv: "f38df8fd4a0883c5231fe4a67d7b50f9"
server_key_version: "encryptionkey@0"
I20260925 02:55:50.089557 24294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:55:50.093339 24309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:50.094677 24294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:55:50.094858 24294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "fa0201da764748a7aad3b07a7ad31338"
format_stamp: "Formatted at 2026-09-25 02:55:50 on dist-test-slave-g1xh"
server_key: "6c05d47ca5d96067d05f60532d4647ef"
server_key_iv: "f38df8fd4a0883c5231fe4a67d7b50f9"
server_key_version: "encryptionkey@0"
I20260925 02:55:50.095067 24294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:50.101022 24294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:50.105064 24294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:50.105365 24294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:50.124642 24294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36501
I20260925 02:55:50.124693 24360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36501 every 8 connection(s)
I20260925 02:55:50.126063 24294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:50.130131 24361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:50.131909 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 24294
I20260925 02:55:50.132090 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:50.132551 2737 external_mini_cluster.cc:1428] Setting key 462ffe568ff34a4dfa754a79076c6dc5
I20260925 02:55:50.141443 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:50.134629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:43937 (local address 127.0.0.1:36501)
0925 02:55:50.135872 (+ 1243us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:50.135889 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:50.135941 (+ 52us) server_negotiation.cc:407] Connection header received
0925 02:55:50.137691 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:50.137713 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:50.138072 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:50.138407 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:50.139754 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:50.139761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:50.139774 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:50.139790 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:50.139849 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:50.139870 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:50.140053 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:50.140208 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 02:55:50.140446 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":658,"threads_started":1}
I20260925 02:55:50.144537 24361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338: Bootstrap starting.
I20260925 02:55:50.145195 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:50.148412 24361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:50.149959 24361 log.cc:824] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:50.153630 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:50.150846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:59813 (local address 127.0.0.1:36501)
0925 02:55:50.151088 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:50.151095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:50.151446 (+ 351us) server_negotiation.cc:407] Connection header received
0925 02:55:50.151688 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:50.151693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:50.151750 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:50.151861 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:50.152910 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:50.152916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:50.152920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:50.152939 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:50.152988 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:50.153008 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:50.153136 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:50.153276 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 02:55:50.153340 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:55:50.153750 24361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338: No bootstrap required, opened a new log
I20260925 02:55:50.161432 24361 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa0201da764748a7aad3b07a7ad31338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36501 } }
I20260925 02:55:50.161830 24361 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:50.161949 24361 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa0201da764748a7aad3b07a7ad31338, State: Initialized, Role: FOLLOWER
I20260925 02:55:50.162602 24361 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa0201da764748a7aad3b07a7ad31338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36501 } }
I20260925 02:55:50.162811 24361 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:50.162972 24361 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:50.163143 24361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:50.165413 24361 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa0201da764748a7aad3b07a7ad31338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36501 } }
I20260925 02:55:50.165922 24361 leader_election.cc:302] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa0201da764748a7aad3b07a7ad31338; no voters:
I20260925 02:55:50.166632 24361 leader_election.cc:288] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:50.166791 24371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:50.168645 24361 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:50.169032 24371 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [term 1 LEADER]: Becoming Leader. State: Replica: fa0201da764748a7aad3b07a7ad31338, State: Running, Role: LEADER
I20260925 02:55:50.169564 24371 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa0201da764748a7aad3b07a7ad31338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36501 } }
I20260925 02:55:50.173259 24373 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa0201da764748a7aad3b07a7ad31338. Latest consensus state: current_term: 1 leader_uuid: "fa0201da764748a7aad3b07a7ad31338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa0201da764748a7aad3b07a7ad31338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36501 } } }
I20260925 02:55:50.173668 24373 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:50.173544 24372 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa0201da764748a7aad3b07a7ad31338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa0201da764748a7aad3b07a7ad31338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36501 } } }
I20260925 02:55:50.174000 24372 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:50.178586 24384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:50.183032 24384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:50.191742 24384 catalog_manager.cc:1347] Generated new cluster ID: dda7f13468e544779b11604886c9627c
I20260925 02:55:50.191875 24384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:50.210430 24384 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:50.210923 24384 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:50.216701 24384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa0201da764748a7aad3b07a7ad31338: Generated new TSK 0
I20260925 02:55:50.217492 24384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:50.223639 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24294
W20260925 02:55:50.224922 24356 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.142:43937
W20260925 02:55:50.225694 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24294: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:55:50.225853 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 24294
2026-09-25T02:55:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:50 dist-test-slave-g1xh krb5kdc[24392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:50 dist-test-slave-g1xh krb5kdc[24392](info): set up 2 sockets
Sep 25 02:55:50 dist-test-slave-g1xh krb5kdc[24392](info): commencing operation
krb5kdc: starting...
W20260925 02:55:51.379284 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.006s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:51 dist-test-slave-g1xh krb5kdc[24392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:55:51.675841 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:52.020991 24405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:52.021410 24405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:52.021590 24405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:52.029228 24405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:52.029412 24405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:52.029531 24405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:52.029645 24405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:52.029724 24405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:52.040212 24405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:52.041704 24405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:52.043186 24405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:52.055446 24410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:52.055404 24411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:52.055377 24413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:52.056186 24405 server_base.cc:1034] running on GCE node
I20260925 02:55:52.057013 24405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:52.058506 24405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:52.059712 24405 hybrid_clock.cc:648] HybridClock initialized: now 1790304952059679 us; error 30 us; skew 500 ppm
I20260925 02:55:52.060086 24405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:52 dist-test-slave-g1xh krb5kdc[24392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:52.063308 24405 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:55:52.064956 24405 webserver.cc:460] Webserver started at http://127.0.0.1:39439/ using document root <none> and password file <none>
I20260925 02:55:52.065670 24405 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:52.065814 24405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:52.066130 24405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:52.069203 24405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "cb269ddeb7a1450bb75fb18b9d657555"
format_stamp: "Formatted at 2026-09-25 02:55:52 on dist-test-slave-g1xh"
server_key: "1bf4e9686ae4ff68b6bae414ff1902ed"
server_key_iv: "f2df7c2b0b51614a5153a777236e34f7"
server_key_version: "encryptionkey@0"
I20260925 02:55:52.069958 24405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "cb269ddeb7a1450bb75fb18b9d657555"
format_stamp: "Formatted at 2026-09-25 02:55:52 on dist-test-slave-g1xh"
server_key: "1bf4e9686ae4ff68b6bae414ff1902ed"
server_key_iv: "f2df7c2b0b51614a5153a777236e34f7"
server_key_version: "encryptionkey@0"
I20260925 02:55:52.075009 24405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:55:52.078450 24420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:52.080055 24405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:55:52.080264 24405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "cb269ddeb7a1450bb75fb18b9d657555"
format_stamp: "Formatted at 2026-09-25 02:55:52 on dist-test-slave-g1xh"
server_key: "1bf4e9686ae4ff68b6bae414ff1902ed"
server_key_iv: "f2df7c2b0b51614a5153a777236e34f7"
server_key_version: "encryptionkey@0"
I20260925 02:55:52.080479 24405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:52.099308 24405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:52.104048 24405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:52.104365 24405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:52.123730 24405 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34091
I20260925 02:55:52.123767 24471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34091 every 8 connection(s)
I20260925 02:55:52.125128 24405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:52.128787 24472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:52.134553 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 24405
I20260925 02:55:52.134763 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:52.135229 2737 external_mini_cluster.cc:1428] Setting key 31dec34240ced5429c90ce3ed53328c7
I20260925 02:55:52.139415 24472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555: Bootstrap starting.
I20260925 02:55:52.143098 24472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:52.144366 24472 log.cc:824] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:52.147643 24472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555: No bootstrap required, opened a new log
Sep 25 02:55:52 dist-test-slave-g1xh krb5kdc[24392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:55:52.156260 24472 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb269ddeb7a1450bb75fb18b9d657555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } }
I20260925 02:55:52.156704 24472 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:52.156823 24472 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb269ddeb7a1450bb75fb18b9d657555, State: Initialized, Role: FOLLOWER
I20260925 02:55:52.157580 24472 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb269ddeb7a1450bb75fb18b9d657555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } }
I20260925 02:55:52.157894 24472 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:52.158063 24472 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:52.158293 24472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:52.160274 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:52.137006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:45395 (local address 127.0.0.1:34091)
0925 02:55:52.137836 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:52.137848 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:52.137888 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:55:52.139666 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:52.139690 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:52.140067 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:52.140372 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:52.142275 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:52.143761 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:52.145488 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:52.146250 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:52.146389 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:52.151349 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:52.151361 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:52.151376 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:52.151418 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:52.155034 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:52.155929 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:52.155936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:52.155942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:52.156121 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:52.156777 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:52.156782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:52.156785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:52.157534 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:52.157892 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:52.158448 (+ 556us) server_negotiation.cc:299] Negotiation successful
0925 02:55:52.158827 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":389,"threads_started":1}
I20260925 02:55:52.161758 24472 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb269ddeb7a1450bb75fb18b9d657555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } }
I20260925 02:55:52.162297 24472 leader_election.cc:302] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb269ddeb7a1450bb75fb18b9d657555; no voters:
I20260925 02:55:52.162981 24472 leader_election.cc:288] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:52.163169 24477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:52.164547 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:52.164489 24477 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [term 1 LEADER]: Becoming Leader. State: Replica: cb269ddeb7a1450bb75fb18b9d657555, State: Running, Role: LEADER
I20260925 02:55:52.165012 24477 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb269ddeb7a1450bb75fb18b9d657555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } }
I20260925 02:55:52.165436 24472 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:52.167814 24479 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb269ddeb7a1450bb75fb18b9d657555. Latest consensus state: current_term: 1 leader_uuid: "cb269ddeb7a1450bb75fb18b9d657555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb269ddeb7a1450bb75fb18b9d657555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } } }
I20260925 02:55:52.168233 24479 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:52.168761 24478 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb269ddeb7a1450bb75fb18b9d657555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb269ddeb7a1450bb75fb18b9d657555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } } }
I20260925 02:55:52.169091 24478 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:52.172258 24486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:52.178272 24486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:52.191711 24486 catalog_manager.cc:1347] Generated new cluster ID: dc4c5a03bc6c4a1da32a7b8ec475c481
I20260925 02:55:52.191912 24486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:52.205863 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:52.175510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:52283 (local address 127.0.0.1:34091)
0925 02:55:52.180188 (+ 4678us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:52.180198 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:52.186498 (+ 6300us) server_negotiation.cc:407] Connection header received
0925 02:55:52.187093 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:52.187102 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:52.187211 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:52.187299 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:52.190012 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:52.191184 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:52.193493 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:52.194298 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:52.194441 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:52.198949 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:52.198959 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:52.198967 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:52.199024 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:52.202670 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:52.203504 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:52.203510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:52.203515 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:52.203721 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:52.204215 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:52.204224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:52.204226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:52.204523 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:52.204807 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:52.205190 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 02:55:52.205372 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4506}
I20260925 02:55:52.212567 24486 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:52.213794 24486 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:52.225670 24486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb269ddeb7a1450bb75fb18b9d657555: Generated new TSK 0
I20260925 02:55:52.226554 24486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:52.235935 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24405
I20260925 02:55:52.237174 24467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:52.330577 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 24405
W20260925 02:55:52.348714 24390 connection.cc:535] client connection to 127.0.0.1:34091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34091): Connection reset by peer (error 104)
2026-09-25T02:55:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2107 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:52 dist-test-slave-g1xh krb5kdc[24504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:52 dist-test-slave-g1xh krb5kdc[24504](info): set up 2 sockets
Sep 25 02:55:52 dist-test-slave-g1xh krb5kdc[24504](info): commencing operation
krb5kdc: starting...
W20260925 02:55:53.474294 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:53 dist-test-slave-g1xh krb5kdc[24504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:55:53.761075 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:54.089861 24517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:54.090179 24517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:54.090370 24517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:54.097957 24517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:54.098085 24517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:54.098186 24517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:54.098302 24517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:54.098387 24517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:54.108641 24517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39581
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:54.110010 24517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:54.111339 24517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:54.123780 24525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:54.123780 24523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:54.124013 24522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:54.124770 24517 server_base.cc:1034] running on GCE node
I20260925 02:55:54.125669 24517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:54.127189 24517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:54.128432 24517 hybrid_clock.cc:648] HybridClock initialized: now 1790304954128391 us; error 35 us; skew 500 ppm
I20260925 02:55:54.128801 24517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:54 dist-test-slave-g1xh krb5kdc[24504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:54.132035 24517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:55:54.133563 24517 webserver.cc:460] Webserver started at http://127.0.0.1:40617/ using document root <none> and password file <none>
I20260925 02:55:54.134269 24517 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:54.134414 24517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:54.134732 24517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:54.137854 24517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "7f6568b537d34a37b8389c7ed85944ca"
format_stamp: "Formatted at 2026-09-25 02:55:54 on dist-test-slave-g1xh"
server_key: "13467a50b8bb743515f7b7a06c38d09a"
server_key_iv: "e3622b8187759e092c17d9a91ac7db47"
server_key_version: "encryptionkey@0"
I20260925 02:55:54.138664 24517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "7f6568b537d34a37b8389c7ed85944ca"
format_stamp: "Formatted at 2026-09-25 02:55:54 on dist-test-slave-g1xh"
server_key: "13467a50b8bb743515f7b7a06c38d09a"
server_key_iv: "e3622b8187759e092c17d9a91ac7db47"
server_key_version: "encryptionkey@0"
I20260925 02:55:54.143931 24517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:55:54.147379 24532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:54.148725 24517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:55:54.148903 24517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "7f6568b537d34a37b8389c7ed85944ca"
format_stamp: "Formatted at 2026-09-25 02:55:54 on dist-test-slave-g1xh"
server_key: "13467a50b8bb743515f7b7a06c38d09a"
server_key_iv: "e3622b8187759e092c17d9a91ac7db47"
server_key_version: "encryptionkey@0"
I20260925 02:55:54.149103 24517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:54.168872 24517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:54.172952 24517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:54.173269 24517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:54.192217 24517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39581
I20260925 02:55:54.192302 24583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39581 every 8 connection(s)
I20260925 02:55:54.193660 24517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:54.197254 24584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:54.197921 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 24517
I20260925 02:55:54.198091 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:54.198541 2737 external_mini_cluster.cc:1428] Setting key 396c507a92915e1f3fdd9d8a4612fab0
I20260925 02:55:54.210466 24584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca: Bootstrap starting.
I20260925 02:55:54.211864 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:54.200340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:55185 (local address 127.0.0.1:39581)
0925 02:55:54.201073 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:54.201084 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:54.201122 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:55:54.203395 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:54.203416 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:54.203884 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:54.204314 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:54.204908 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:54.206134 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:54.207429 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:54.208182 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:54.208410 (+ 228us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:54.209652 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:54.209660 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:54.209673 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:54.210191 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:54.210272 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:54.210297 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:54.210473 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:54.210700 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 02:55:54.210877 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":323,"threads_started":1}
I20260925 02:55:54.214522 24584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:54.215862 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:54.215924 24584 log.cc:824] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:54.218971 24584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca: No bootstrap required, opened a new log
I20260925 02:55:54.227145 24584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6568b537d34a37b8389c7ed85944ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39581 } }
I20260925 02:55:54.227783 24584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:54.227917 24584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6568b537d34a37b8389c7ed85944ca, State: Initialized, Role: FOLLOWER
I20260925 02:55:54.228646 24584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6568b537d34a37b8389c7ed85944ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39581 } }
I20260925 02:55:54.228928 24584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:54.229123 24584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:54.229317 24584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:54.232827 24584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6568b537d34a37b8389c7ed85944ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39581 } }
I20260925 02:55:54.233491 24584 leader_election.cc:302] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f6568b537d34a37b8389c7ed85944ca; no voters:
I20260925 02:55:54.233757 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:54.220288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:51879 (local address 127.0.0.1:39581)
0925 02:55:54.220540 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:54.220550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:54.226792 (+ 6242us) server_negotiation.cc:407] Connection header received
0925 02:55:54.227011 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:54.227017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:54.227098 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:54.227205 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:54.227995 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:54.228787 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:54.230922 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:54.231379 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:54.231472 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:54.232833 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:54.232838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:54.232841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:54.232858 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:54.232909 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:54.232925 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:54.233050 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:54.233305 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 02:55:54.233371 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:55:54.234288 24584 leader_election.cc:288] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:54.234428 24594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:54.236516 24584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:54.240715 24594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [term 1 LEADER]: Becoming Leader. State: Replica: 7f6568b537d34a37b8389c7ed85944ca, State: Running, Role: LEADER
I20260925 02:55:54.241398 24594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6568b537d34a37b8389c7ed85944ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39581 } }
I20260925 02:55:54.244747 24595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f6568b537d34a37b8389c7ed85944ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6568b537d34a37b8389c7ed85944ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39581 } } }
I20260925 02:55:54.245251 24595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:54.244993 24596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f6568b537d34a37b8389c7ed85944ca. Latest consensus state: current_term: 1 leader_uuid: "7f6568b537d34a37b8389c7ed85944ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6568b537d34a37b8389c7ed85944ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39581 } } }
I20260925 02:55:54.245388 24596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca [sys.catalog]: This master's current role is: LEADER
W20260925 02:55:54.250151 24609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:55:54.250300 24609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:55:54.250445 24610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:54.252374 24610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:54.260594 24610 catalog_manager.cc:1347] Generated new cluster ID: d2ac6bbf00cb4c8cb09d394d47dcd65a
I20260925 02:55:54.260712 24610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:54.275538 24610 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:54.276803 24610 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:54.287725 24610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f6568b537d34a37b8389c7ed85944ca: Generated new TSK 0
I20260925 02:55:54.288517 24610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:54.296238 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24517
W20260925 02:55:54.297595 24579 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.142:55185
W20260925 02:55:54.298491 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24517: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:55:54.298688 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 24517
W20260925 02:55:54.316342 24502 connection.cc:535] client connection to 127.0.0.1:39581 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39581): Connection reset by peer (error 104)
2026-09-25T02:55:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1967 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:54 dist-test-slave-g1xh krb5kdc[24616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:54 dist-test-slave-g1xh krb5kdc[24616](info): set up 2 sockets
Sep 25 02:55:54 dist-test-slave-g1xh krb5kdc[24616](info): commencing operation
krb5kdc: starting...
W20260925 02:55:55.434280 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:55 dist-test-slave-g1xh krb5kdc[24616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:55:55.723018 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:56.051676 24629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:56.052066 24629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:56.052235 24629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:56.060070 24629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:56.060232 24629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:56.060346 24629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:56.060456 24629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:56.060534 24629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:56.070770 24629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:56.072193 24629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:56.073534 24629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:56.085832 24637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:56.085979 24634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:56.085947 24635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:56.086719 24629 server_base.cc:1034] running on GCE node
I20260925 02:55:56.087451 24629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:56.088984 24629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:56.090191 24629 hybrid_clock.cc:648] HybridClock initialized: now 1790304956090153 us; error 42 us; skew 500 ppm
I20260925 02:55:56.090588 24629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:56 dist-test-slave-g1xh krb5kdc[24616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:56.093956 24629 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:55:56.095456 24629 webserver.cc:460] Webserver started at http://127.0.0.1:41741/ using document root <none> and password file <none>
I20260925 02:55:56.096176 24629 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:56.096304 24629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:56.096560 24629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:56.099762 24629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "b436954dba8b49d68118557e583a2572"
format_stamp: "Formatted at 2026-09-25 02:55:56 on dist-test-slave-g1xh"
server_key: "1ef2ed466f5c759938afea0a6fc67cec"
server_key_iv: "df280c8b106bdf7b68d89a03e6781ec4"
server_key_version: "encryptionkey@0"
I20260925 02:55:56.100494 24629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "b436954dba8b49d68118557e583a2572"
format_stamp: "Formatted at 2026-09-25 02:55:56 on dist-test-slave-g1xh"
server_key: "1ef2ed466f5c759938afea0a6fc67cec"
server_key_iv: "df280c8b106bdf7b68d89a03e6781ec4"
server_key_version: "encryptionkey@0"
I20260925 02:55:56.105967 24629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 02:55:56.109680 24644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:56.111068 24629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:55:56.111266 24629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "b436954dba8b49d68118557e583a2572"
format_stamp: "Formatted at 2026-09-25 02:55:56 on dist-test-slave-g1xh"
server_key: "1ef2ed466f5c759938afea0a6fc67cec"
server_key_iv: "df280c8b106bdf7b68d89a03e6781ec4"
server_key_version: "encryptionkey@0"
I20260925 02:55:56.111478 24629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:56.117439 24629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:56.122228 24629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:56.122545 24629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:56.145280 24629 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35885
I20260925 02:55:56.145372 24696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35885 every 8 connection(s)
I20260925 02:55:56.146775 24629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:56.150352 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 24629
I20260925 02:55:56.150573 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:56.150892 24697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:56.151127 2737 external_mini_cluster.cc:1428] Setting key 34d8c76c45765fb31285c02045ec56c6
I20260925 02:55:56.161026 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:56.153390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:58457 (local address 127.0.0.1:35885)
0925 02:55:56.154769 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:56.154781 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:56.154826 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:55:56.157346 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:56.157378 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:56.157769 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:56.158136 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:56.159339 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:56.159346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:56.159360 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:56.159375 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:56.159438 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:56.159458 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:56.159725 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:56.159835 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 02:55:56.160063 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":604,"threads_started":1}
I20260925 02:55:56.164793 24697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572: Bootstrap starting.
I20260925 02:55:56.164981 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:56.168785 24697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:56.170243 24697 log.cc:824] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:56.174345 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:56.170620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:43855 (local address 127.0.0.1:35885)
0925 02:55:56.170780 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:56.170785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:56.171141 (+ 356us) server_negotiation.cc:407] Connection header received
0925 02:55:56.171292 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:56.171296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:56.171363 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:56.171489 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:56.173537 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:56.173542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:56.173545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:55:56.173560 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:56.173607 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:55:56.173621 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:56.173735 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:56.173884 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 02:55:56.173937 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:55:56.174890 24697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572: No bootstrap required, opened a new log
I20260925 02:55:56.182379 24697 raft_consensus.cc:348] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b436954dba8b49d68118557e583a2572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260925 02:55:56.182821 24697 raft_consensus.cc:374] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:56.182935 24697 raft_consensus.cc:729] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b436954dba8b49d68118557e583a2572, State: Initialized, Role: FOLLOWER
I20260925 02:55:56.183513 24697 consensus_queue.cc:260] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b436954dba8b49d68118557e583a2572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260925 02:55:56.183876 24697 raft_consensus.cc:388] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:56.184024 24697 raft_consensus.cc:482] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:56.184201 24697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:56.186435 24697 raft_consensus.cc:504] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b436954dba8b49d68118557e583a2572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260925 02:55:56.186957 24697 leader_election.cc:302] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b436954dba8b49d68118557e583a2572; no voters:
I20260925 02:55:56.187667 24697 leader_election.cc:288] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:56.187793 24707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:56.188621 24707 raft_consensus.cc:686] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [term 1 LEADER]: Becoming Leader. State: Replica: b436954dba8b49d68118557e583a2572, State: Running, Role: LEADER
I20260925 02:55:56.189217 24707 consensus_queue.cc:237] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b436954dba8b49d68118557e583a2572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260925 02:55:56.190093 24697 sys_catalog.cc:564] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:56.192225 24708 sys_catalog.cc:455] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b436954dba8b49d68118557e583a2572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b436954dba8b49d68118557e583a2572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } } }
I20260925 02:55:56.192613 24708 sys_catalog.cc:458] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:56.193615 24709 sys_catalog.cc:455] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b436954dba8b49d68118557e583a2572. Latest consensus state: current_term: 1 leader_uuid: "b436954dba8b49d68118557e583a2572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b436954dba8b49d68118557e583a2572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } } }
I20260925 02:55:56.194581 24709 sys_catalog.cc:458] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:56.195058 24713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:56.200932 24713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:56.209651 24713 catalog_manager.cc:1347] Generated new cluster ID: f8e5d132f77e4f2799cc50aa19913aeb
I20260925 02:55:56.209777 24713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:56.230686 24713 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:56.231179 24713 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:56.247223 24713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b436954dba8b49d68118557e583a2572: Generated new TSK 0
I20260925 02:55:56.248355 24713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:56.257757 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24629
W20260925 02:55:56.258841 24692 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.142:58457
W20260925 02:55:56.259533 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24629: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:55:56.259716 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 24629
2026-09-25T02:55:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:56 dist-test-slave-g1xh krb5kdc[24728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:56 dist-test-slave-g1xh krb5kdc[24728](info): set up 2 sockets
Sep 25 02:55:56 dist-test-slave-g1xh krb5kdc[24728](info): commencing operation
krb5kdc: starting...
W20260925 02:55:57.394025 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:57 dist-test-slave-g1xh krb5kdc[24728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:55:57.687247 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:36249
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:36249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:58.022114 24741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:58.022403 24741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:58.022500 24741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:58.029799 24741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:58.029919 24741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:58.029968 24741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:58.030032 24741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:58.030077 24741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:58.040186 24741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:36249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:36249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:55:58.041433 24741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:58.042726 24741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:58.054685 24746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:58.054654 24749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:58.054729 24747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:58.055891 24741 server_base.cc:1034] running on GCE node
I20260925 02:55:58.056654 24741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:58.058063 24741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:58.059271 24741 hybrid_clock.cc:648] HybridClock initialized: now 1790304958059230 us; error 34 us; skew 500 ppm
I20260925 02:55:58.059672 24741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:55:58 dist-test-slave-g1xh krb5kdc[24728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304958, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:58.062909 24741 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:55:58.064536 24741 webserver.cc:460] Webserver started at http://127.2.172.126:42629/ using document root <none> and password file <none>
I20260925 02:55:58.065291 24741 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:58.065403 24741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:58.065668 24741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:58.068837 24741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "9f2a107870844d769e743e0c018eee06"
format_stamp: "Formatted at 2026-09-25 02:55:58 on dist-test-slave-g1xh"
server_key: "6584d96b31bfef93067be2cb03a0d430"
server_key_iv: "033a5d92763d6cf0265eb59ee71bd908"
server_key_version: "encryptionkey@0"
I20260925 02:55:58.069584 24741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "9f2a107870844d769e743e0c018eee06"
format_stamp: "Formatted at 2026-09-25 02:55:58 on dist-test-slave-g1xh"
server_key: "6584d96b31bfef93067be2cb03a0d430"
server_key_iv: "033a5d92763d6cf0265eb59ee71bd908"
server_key_version: "encryptionkey@0"
I20260925 02:55:58.075663 24741 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 02:55:58.080117 24756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:58.082042 24741 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260925 02:55:58.082218 24741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "9f2a107870844d769e743e0c018eee06"
format_stamp: "Formatted at 2026-09-25 02:55:58 on dist-test-slave-g1xh"
server_key: "6584d96b31bfef93067be2cb03a0d430"
server_key_iv: "033a5d92763d6cf0265eb59ee71bd908"
server_key_version: "encryptionkey@0"
I20260925 02:55:58.082409 24741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:58.108752 24741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:58.113193 24741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:58.113446 24741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:58.132547 24741 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:36249
I20260925 02:55:58.132599 24807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:36249 every 8 connection(s)
I20260925 02:55:58.133932 24741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:58.136845 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 24741
I20260925 02:55:58.137035 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:55:58.137566 2737 external_mini_cluster.cc:1428] Setting key 4faef3411b95c5b92c51c8e1298afe1a
I20260925 02:55:58.138109 24808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:58.150453 24808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06: Bootstrap starting.
Sep 25 02:55:58 dist-test-slave-g1xh krb5kdc[24728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:55:58.154587 24808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:58.155995 24808 log.cc:824] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:58.159648 24808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06: No bootstrap required, opened a new log
I20260925 02:55:58.160740 24811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:58.139375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42595 (local address 127.2.172.126:36249)
0925 02:55:58.140147 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:58.140159 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:58.140214 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:55:58.141721 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:58.141741 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:58.142114 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:58.142525 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:58.144020 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:58.145357 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:58.146616 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:58.147266 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:58.147484 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:58.151645 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:58.151658 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:58.151676 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:58.151725 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:58.155857 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:58.156757 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:58.156764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:58.156770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:58.156937 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:58.157486 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:58.157491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:58.157495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:58.158202 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:58.158600 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:58.158921 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 02:55:58.159255 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":357,"threads_started":1}
I20260925 02:55:58.164870 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:58.168551 24808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2a107870844d769e743e0c018eee06" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36249 } }
I20260925 02:55:58.169057 24808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:58.169201 24808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2a107870844d769e743e0c018eee06, State: Initialized, Role: FOLLOWER
I20260925 02:55:58.169975 24808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2a107870844d769e743e0c018eee06" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36249 } }
I20260925 02:55:58.170239 24808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:58.170409 24808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:58.170588 24808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:58.173022 24808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2a107870844d769e743e0c018eee06" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36249 } }
I20260925 02:55:58.173549 24808 leader_election.cc:302] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f2a107870844d769e743e0c018eee06; no voters:
I20260925 02:55:58.174261 24808 leader_election.cc:288] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:58.174376 24819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:58.175820 24819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [term 1 LEADER]: Becoming Leader. State: Replica: 9f2a107870844d769e743e0c018eee06, State: Running, Role: LEADER
I20260925 02:55:58.176492 24819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2a107870844d769e743e0c018eee06" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36249 } }
I20260925 02:55:58.176844 24808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:58.180171 24820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f2a107870844d769e743e0c018eee06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2a107870844d769e743e0c018eee06" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36249 } } }
I20260925 02:55:58.181093 24820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:58.185707 24821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f2a107870844d769e743e0c018eee06. Latest consensus state: current_term: 1 leader_uuid: "9f2a107870844d769e743e0c018eee06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2a107870844d769e743e0c018eee06" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36249 } } }
I20260925 02:55:58.186857 24821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:58.188326 24811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:58.169331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51295 (local address 127.2.172.126:36249)
0925 02:55:58.169507 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:58.169513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:58.169865 (+ 352us) server_negotiation.cc:407] Connection header received
0925 02:55:58.170058 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:58.170064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:58.170137 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:58.170254 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:58.171854 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:58.172644 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:58.173985 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:58.174598 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:58.174773 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:58.177810 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:58.177816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:58.177821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:58.177868 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:58.182411 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:58.183867 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:58.183875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:58.183881 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:58.184093 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:58.184866 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:58.184874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:58.184881 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:58.185380 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:58.187272 (+ 1892us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:58.187502 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 02:55:58.187771 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:55:58.192668 24827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:58.198421 24827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:58.208165 24827 catalog_manager.cc:1347] Generated new cluster ID: 76de2de61471496a8f80b11f74a5ca71
I20260925 02:55:58.208292 24827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:58.245716 24827 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:58.247414 24827 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:58.253396 24827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f2a107870844d769e743e0c018eee06: Generated new TSK 0
I20260925 02:55:58.254405 24827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:58.259896 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24741
I20260925 02:55:58.261313 24803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:55:58.354377 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 24741
W20260925 02:55:58.372426 24726 connection.cc:535] client connection to 127.2.172.126:36249 recv error: Network error: failed to read from TLS socket (remote: 127.2.172.126:36249): Connection reset by peer (error 104)
2026-09-25T02:55:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2095 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:58 dist-test-slave-g1xh krb5kdc[24841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:58 dist-test-slave-g1xh krb5kdc[24841](info): set up 2 sockets
Sep 25 02:55:58 dist-test-slave-g1xh krb5kdc[24841](info): commencing operation
krb5kdc: starting...
W20260925 02:55:59.474181 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:55:59 dist-test-slave-g1xh krb5kdc[24841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:55:59.759057 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:35161
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:35161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:00.095276 24854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:00.095649 24854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:00.095753 24854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:00.103170 24854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:00.103308 24854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:00.103356 24854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:00.103442 24854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:00.103492 24854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:00.113889 24854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:35161
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:35161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:00.115302 24854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:00.116663 24854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:00.128463 24859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:00.128554 24860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:00.128579 24862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:00.129567 24854 server_base.cc:1034] running on GCE node
I20260925 02:56:00.130291 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:00.131629 24854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:00.132838 24854 hybrid_clock.cc:648] HybridClock initialized: now 1790304960132789 us; error 49 us; skew 500 ppm
I20260925 02:56:00.133124 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:00 dist-test-slave-g1xh krb5kdc[24841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304960, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:00.136276 24854 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:56:00.137686 24854 webserver.cc:460] Webserver started at http://127.2.172.126:35625/ using document root <none> and password file <none>
I20260925 02:56:00.138288 24854 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:00.138391 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:00.138643 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:00.141575 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "f113b4aa13c040a09fac9db3c58ceb27"
format_stamp: "Formatted at 2026-09-25 02:56:00 on dist-test-slave-g1xh"
server_key: "abc1b379c53dfae4c370de58504b4dea"
server_key_iv: "a472136e05316f5e0d2db16f7277a70c"
server_key_version: "encryptionkey@0"
I20260925 02:56:00.142298 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "f113b4aa13c040a09fac9db3c58ceb27"
format_stamp: "Formatted at 2026-09-25 02:56:00 on dist-test-slave-g1xh"
server_key: "abc1b379c53dfae4c370de58504b4dea"
server_key_iv: "a472136e05316f5e0d2db16f7277a70c"
server_key_version: "encryptionkey@0"
I20260925 02:56:00.147200 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 02:56:00.150543 24869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:00.151939 24854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:56:00.152077 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "f113b4aa13c040a09fac9db3c58ceb27"
format_stamp: "Formatted at 2026-09-25 02:56:00 on dist-test-slave-g1xh"
server_key: "abc1b379c53dfae4c370de58504b4dea"
server_key_iv: "a472136e05316f5e0d2db16f7277a70c"
server_key_version: "encryptionkey@0"
I20260925 02:56:00.152245 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:00.184226 24854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:00.188905 24854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:00.189160 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:00.208511 24854 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:35161
I20260925 02:56:00.208559 24920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:35161 every 8 connection(s)
I20260925 02:56:00.209954 24854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:00.213747 24921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:00.216706 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 24854
I20260925 02:56:00.216892 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:56:00.217432 2737 external_mini_cluster.cc:1428] Setting key 81eb9953ef17d0cee95af4727a6167c0
I20260925 02:56:00.226086 24921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27: Bootstrap starting.
I20260925 02:56:00.229754 24921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:00.230635 24924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:00.219247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52069 (local address 127.2.172.126:35161)
0925 02:56:00.220062 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:00.220073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:00.220112 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:56:00.221736 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:00.221758 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:00.222151 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:00.222469 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:00.223055 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:00.224482 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:00.226317 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:00.226945 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:00.227068 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:00.228435 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:00.228443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:00.228455 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:00.228957 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:00.229028 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:00.229052 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:00.229168 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:00.229436 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 02:56:00.229621 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":403,"threads_started":1}
I20260925 02:56:00.230878 24921 log.cc:824] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:00.234050 24921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27: No bootstrap required, opened a new log
I20260925 02:56:00.234709 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:00.244735 24921 raft_consensus.cc:348] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f113b4aa13c040a09fac9db3c58ceb27" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 35161 } }
I20260925 02:56:00.245252 24921 raft_consensus.cc:374] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:00.245366 24921 raft_consensus.cc:729] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f113b4aa13c040a09fac9db3c58ceb27, State: Initialized, Role: FOLLOWER
I20260925 02:56:00.246040 24921 consensus_queue.cc:260] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f113b4aa13c040a09fac9db3c58ceb27" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 35161 } }
I20260925 02:56:00.246299 24921 raft_consensus.cc:388] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:00.246512 24921 raft_consensus.cc:482] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:00.246588 24924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:00.239917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56071 (local address 127.2.172.126:35161)
0925 02:56:00.240103 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:00.240110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:00.240629 (+ 519us) server_negotiation.cc:407] Connection header received
0925 02:56:00.240766 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:00.240771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:00.240852 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:00.240965 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:00.241409 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:00.242631 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:00.243990 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:00.244621 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:00.244774 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:00.245831 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:00.245843 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:00.245846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:00.245864 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:00.245917 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:00.245932 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:00.246061 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:00.246257 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 02:56:00.246313 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:56:00.246752 24921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:00.249549 24921 raft_consensus.cc:504] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f113b4aa13c040a09fac9db3c58ceb27" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 35161 } }
I20260925 02:56:00.250057 24921 leader_election.cc:302] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f113b4aa13c040a09fac9db3c58ceb27; no voters:
I20260925 02:56:00.250639 24921 leader_election.cc:288] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:00.250782 24931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:00.251593 24931 raft_consensus.cc:686] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [term 1 LEADER]: Becoming Leader. State: Replica: f113b4aa13c040a09fac9db3c58ceb27, State: Running, Role: LEADER
I20260925 02:56:00.252218 24931 consensus_queue.cc:237] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f113b4aa13c040a09fac9db3c58ceb27" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 35161 } }
I20260925 02:56:00.253082 24921 sys_catalog.cc:564] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:00.255245 24933 sys_catalog.cc:455] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f113b4aa13c040a09fac9db3c58ceb27. Latest consensus state: current_term: 1 leader_uuid: "f113b4aa13c040a09fac9db3c58ceb27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f113b4aa13c040a09fac9db3c58ceb27" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 35161 } } }
I20260925 02:56:00.255774 24933 sys_catalog.cc:458] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:00.256004 24932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f113b4aa13c040a09fac9db3c58ceb27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f113b4aa13c040a09fac9db3c58ceb27" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 35161 } } }
I20260925 02:56:00.256421 24932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:00.258195 24939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:00.263682 24939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:00.272763 24939 catalog_manager.cc:1347] Generated new cluster ID: fbd99db1231e4cdaa2e3dc06d9d5427d
I20260925 02:56:00.272903 24939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:00.291949 24939 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:00.293287 24939 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:00.316128 24939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f113b4aa13c040a09fac9db3c58ceb27: Generated new TSK 0
I20260925 02:56:00.317014 24939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:00.323351 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24854
W20260925 02:56:00.325166 24916 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52069
W20260925 02:56:00.325984 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24854: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:00.326154 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 24854
W20260925 02:56:00.344604 24839 connection.cc:535] client connection to 127.2.172.126:35161 recv error: Network error: failed to read from TLS socket (remote: 127.2.172.126:35161): Connection reset by peer (error 104)
2026-09-25T02:56:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1972 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:00 dist-test-slave-g1xh krb5kdc[24953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:00 dist-test-slave-g1xh krb5kdc[24953](info): set up 2 sockets
Sep 25 02:56:00 dist-test-slave-g1xh krb5kdc[24953](info): commencing operation
krb5kdc: starting...
W20260925 02:56:01.446197 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.007s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:56:01 dist-test-slave-g1xh krb5kdc[24953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:56:01.744289 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:34753
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:34753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:02.083285 24966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:02.083652 24966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:02.083765 24966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:02.091217 24966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:02.091338 24966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:02.091394 24966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:02.091511 24966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:02.091616 24966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:02.101974 24966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:34753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:34753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:02.103286 24966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:02.104686 24966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:02.117447 24971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:02.117434 24972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:02.117421 24974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:02.117993 24966 server_base.cc:1034] running on GCE node
I20260925 02:56:02.119055 24966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:02.120738 24966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:02.122018 24966 hybrid_clock.cc:648] HybridClock initialized: now 1790304962121961 us; error 37 us; skew 500 ppm
I20260925 02:56:02.122424 24966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:02 dist-test-slave-g1xh krb5kdc[24953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304962, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:02.125777 24966 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:56:02.127339 24966 webserver.cc:460] Webserver started at http://127.2.172.126:37633/ using document root <none> and password file <none>
I20260925 02:56:02.128135 24966 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:02.128298 24966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:02.128641 24966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:02.132018 24966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "e107fdc8079c4beb92ddf47213926014"
format_stamp: "Formatted at 2026-09-25 02:56:02 on dist-test-slave-g1xh"
server_key: "976e04451fd7f3ddac57f95c88c5e7f2"
server_key_iv: "03643574df3b5b4f5540f49ad8ecfbd2"
server_key_version: "encryptionkey@0"
I20260925 02:56:02.132885 24966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "e107fdc8079c4beb92ddf47213926014"
format_stamp: "Formatted at 2026-09-25 02:56:02 on dist-test-slave-g1xh"
server_key: "976e04451fd7f3ddac57f95c88c5e7f2"
server_key_iv: "03643574df3b5b4f5540f49ad8ecfbd2"
server_key_version: "encryptionkey@0"
I20260925 02:56:02.138423 24966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:56:02.142263 24981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:02.143709 24966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:56:02.143888 24966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "e107fdc8079c4beb92ddf47213926014"
format_stamp: "Formatted at 2026-09-25 02:56:02 on dist-test-slave-g1xh"
server_key: "976e04451fd7f3ddac57f95c88c5e7f2"
server_key_iv: "03643574df3b5b4f5540f49ad8ecfbd2"
server_key_version: "encryptionkey@0"
I20260925 02:56:02.144119 24966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:02.149699 24966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:02.153666 24966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:02.153961 24966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:02.172744 24966 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:34753
I20260925 02:56:02.172837 25032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:34753 every 8 connection(s)
I20260925 02:56:02.174221 24966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:02.178130 25033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:02.183352 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 24966
I20260925 02:56:02.183540 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:56:02.184073 2737 external_mini_cluster.cc:1428] Setting key bd442e6f35fdd9f7867dd376a2efcdd8
I20260925 02:56:02.190848 25033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014: Bootstrap starting.
I20260925 02:56:02.193380 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:02.185911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49383 (local address 127.2.172.126:34753)
0925 02:56:02.186776 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:02.186793 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:02.186849 (+ 56us) server_negotiation.cc:407] Connection header received
0925 02:56:02.189418 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:02.189449 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:02.189904 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:02.190367 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:02.191385 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:02.191393 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:02.191416 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:02.191434 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:02.191503 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:02.191529 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:02.191764 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:02.191895 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 02:56:02.192094 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":347,"threads_started":1}
I20260925 02:56:02.194784 25033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:02.196177 25033 log.cc:824] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:02.197748 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:02.199492 25033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014: No bootstrap required, opened a new log
I20260925 02:56:02.204856 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:02.202179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51243 (local address 127.2.172.126:34753)
0925 02:56:02.202422 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:02.202428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:02.202798 (+ 370us) server_negotiation.cc:407] Connection header received
0925 02:56:02.202963 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:02.202967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:02.203013 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:02.203100 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:02.204169 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:02.204173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:02.204176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:02.204191 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:02.204236 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:02.204254 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:02.204345 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:02.204489 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:56:02.204546 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:56:02.208623 25033 raft_consensus.cc:348] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e107fdc8079c4beb92ddf47213926014" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34753 } }
I20260925 02:56:02.209007 25033 raft_consensus.cc:374] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:02.209111 25033 raft_consensus.cc:729] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e107fdc8079c4beb92ddf47213926014, State: Initialized, Role: FOLLOWER
I20260925 02:56:02.209712 25033 consensus_queue.cc:260] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e107fdc8079c4beb92ddf47213926014" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34753 } }
I20260925 02:56:02.210050 25033 raft_consensus.cc:388] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:02.210216 25033 raft_consensus.cc:482] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:02.210413 25033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:02.212787 25033 raft_consensus.cc:504] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e107fdc8079c4beb92ddf47213926014" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34753 } }
I20260925 02:56:02.213223 25033 leader_election.cc:302] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e107fdc8079c4beb92ddf47213926014; no voters:
I20260925 02:56:02.213801 25033 leader_election.cc:288] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:02.213965 25043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:02.214635 25043 raft_consensus.cc:686] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [term 1 LEADER]: Becoming Leader. State: Replica: e107fdc8079c4beb92ddf47213926014, State: Running, Role: LEADER
I20260925 02:56:02.215219 25043 consensus_queue.cc:237] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e107fdc8079c4beb92ddf47213926014" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34753 } }
I20260925 02:56:02.215951 25033 sys_catalog.cc:564] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:02.218245 25044 sys_catalog.cc:455] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e107fdc8079c4beb92ddf47213926014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e107fdc8079c4beb92ddf47213926014" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34753 } } }
I20260925 02:56:02.218415 25045 sys_catalog.cc:455] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e107fdc8079c4beb92ddf47213926014. Latest consensus state: current_term: 1 leader_uuid: "e107fdc8079c4beb92ddf47213926014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e107fdc8079c4beb92ddf47213926014" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 34753 } } }
I20260925 02:56:02.218673 25044 sys_catalog.cc:458] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:02.218700 25045 sys_catalog.cc:458] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:02.219617 25051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:02.224840 25051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:02.233021 25051 catalog_manager.cc:1347] Generated new cluster ID: edbec183d1164aca912f4fb3e0062104
I20260925 02:56:02.233140 25051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:02.256685 25051 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:02.257146 25051 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:02.267004 25051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e107fdc8079c4beb92ddf47213926014: Generated new TSK 0
I20260925 02:56:02.267889 25051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:02.280849 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24966
W20260925 02:56:02.281905 25028 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49383
W20260925 02:56:02.282596 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 24966: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:02.282730 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 24966
2026-09-25T02:56:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1953 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:02 dist-test-slave-g1xh krb5kdc[25064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:02 dist-test-slave-g1xh krb5kdc[25064](info): set up 2 sockets
Sep 25 02:56:02 dist-test-slave-g1xh krb5kdc[25064](info): commencing operation
krb5kdc: starting...
W20260925 02:56:03.402141 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:56:03 dist-test-slave-g1xh krb5kdc[25064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:56:03.715291 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:36225
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:04.051367 25077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:04.051666 25077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:04.051846 25077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:04.059044 25077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:04.059176 25077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:04.059262 25077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:04.059361 25077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:04.059437 25077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:04.069141 25077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:36225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:36225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:04.070472 25077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:04.071810 25077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:04.083459 25083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:04.083703 25082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:04.083823 25085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:04.084913 25077 server_base.cc:1034] running on GCE node
I20260925 02:56:04.085661 25077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:04.087060 25077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:04.088359 25077 hybrid_clock.cc:648] HybridClock initialized: now 1790304964088330 us; error 59 us; skew 500 ppm
I20260925 02:56:04.088791 25077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:04 dist-test-slave-g1xh krb5kdc[25064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304964, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:04.096547 25077 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:56:04.098178 25077 webserver.cc:460] Webserver started at http://127.2.172.126:42389/ using document root <none> and password file <none>
I20260925 02:56:04.098869 25077 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:04.098975 25077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:04.099238 25077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:04.102306 25077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "21701be1fb17446f8881ce8b68bc84a3"
format_stamp: "Formatted at 2026-09-25 02:56:04 on dist-test-slave-g1xh"
server_key: "7f0a417293937178dc02d59bf33fdff0"
server_key_iv: "391dd3a0f367002fb3ebc7d26efa83a7"
server_key_version: "encryptionkey@0"
I20260925 02:56:04.103039 25077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "21701be1fb17446f8881ce8b68bc84a3"
format_stamp: "Formatted at 2026-09-25 02:56:04 on dist-test-slave-g1xh"
server_key: "7f0a417293937178dc02d59bf33fdff0"
server_key_iv: "391dd3a0f367002fb3ebc7d26efa83a7"
server_key_version: "encryptionkey@0"
I20260925 02:56:04.108202 25077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:56:04.111662 25092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:04.112998 25077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:56:04.113179 25077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "21701be1fb17446f8881ce8b68bc84a3"
format_stamp: "Formatted at 2026-09-25 02:56:04 on dist-test-slave-g1xh"
server_key: "7f0a417293937178dc02d59bf33fdff0"
server_key_iv: "391dd3a0f367002fb3ebc7d26efa83a7"
server_key_version: "encryptionkey@0"
I20260925 02:56:04.113370 25077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:04.139354 25077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:04.143651 25077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:04.143934 25077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:04.162691 25077 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:36225
I20260925 02:56:04.162711 25144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:36225 every 8 connection(s)
I20260925 02:56:04.164093 25077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:04.167872 25145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:04.172586 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 25077
I20260925 02:56:04.172778 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:56:04.173377 2737 external_mini_cluster.cc:1428] Setting key 55206b58b9b95b52f628ffb1d915f5da
I20260925 02:56:04.179785 25145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3: Bootstrap starting.
I20260925 02:56:04.183542 25145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:04.184700 25145 log.cc:824] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:04.188279 25145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3: No bootstrap required, opened a new log
Sep 25 02:56:04 dist-test-slave-g1xh krb5kdc[25064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:56:04.196516 25145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21701be1fb17446f8881ce8b68bc84a3" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36225 } }
I20260925 02:56:04.196961 25145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:04.197074 25145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21701be1fb17446f8881ce8b68bc84a3, State: Initialized, Role: FOLLOWER
I20260925 02:56:04.197417 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:04.175237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48271 (local address 127.2.172.126:36225)
0925 02:56:04.176024 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:04.176040 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:04.176084 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:56:04.178091 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:04.178122 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:04.178552 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:04.178894 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:04.180745 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:04.182068 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:04.183877 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:04.184708 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:04.184901 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:04.189794 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:04.189802 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:04.189817 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:04.189858 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:04.193123 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:04.193845 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:04.193853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:04.193860 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:04.194020 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:04.194454 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:04.194464 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:04.194466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:04.195009 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:04.195335 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:04.195781 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 02:56:04.196238 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":379,"threads_started":1}
I20260925 02:56:04.197767 25145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21701be1fb17446f8881ce8b68bc84a3" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36225 } }
I20260925 02:56:04.198086 25145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:04.198232 25145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:04.198405 25145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:04.201418 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:04.201530 25145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21701be1fb17446f8881ce8b68bc84a3" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36225 } }
I20260925 02:56:04.202006 25145 leader_election.cc:302] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21701be1fb17446f8881ce8b68bc84a3; no voters:
I20260925 02:56:04.202666 25145 leader_election.cc:288] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:04.202764 25150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:04.205283 25145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:04.208281 25150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [term 1 LEADER]: Becoming Leader. State: Replica: 21701be1fb17446f8881ce8b68bc84a3, State: Running, Role: LEADER
I20260925 02:56:04.209000 25150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21701be1fb17446f8881ce8b68bc84a3" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36225 } }
I20260925 02:56:04.212224 25151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21701be1fb17446f8881ce8b68bc84a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21701be1fb17446f8881ce8b68bc84a3" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36225 } } }
I20260925 02:56:04.212683 25151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:04.213161 25152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21701be1fb17446f8881ce8b68bc84a3. Latest consensus state: current_term: 1 leader_uuid: "21701be1fb17446f8881ce8b68bc84a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21701be1fb17446f8881ce8b68bc84a3" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36225 } } }
I20260925 02:56:04.213474 25152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:04.216972 25160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:04.222530 25160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:04.234937 25160 catalog_manager.cc:1347] Generated new cluster ID: 3a746bf1a7e24ae7b2260846edc47159
I20260925 02:56:04.235044 25160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:04.237017 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:04.215922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44965 (local address 127.2.172.126:36225)
0925 02:56:04.216272 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:04.216282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:04.217207 (+ 925us) server_negotiation.cc:407] Connection header received
0925 02:56:04.217398 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:04.217404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:04.217515 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:04.217653 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:04.220284 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:04.221776 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:04.224375 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:04.225338 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:04.225480 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:04.229453 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:04.229467 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:04.229475 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:04.229528 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:04.233768 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:04.234688 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:04.234696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:04.234700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:04.234848 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:04.235341 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:04.235346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:04.235349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:04.235788 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:04.236054 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:04.236461 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 02:56:04.236696 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260925 02:56:04.243306 25160 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:04.245147 25160 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:04.257282 25160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21701be1fb17446f8881ce8b68bc84a3: Generated new TSK 0
I20260925 02:56:04.258417 25160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:04.268535 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 25077
I20260925 02:56:04.269588 25140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:04.358536 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 25077
2026-09-25T02:56:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:04 dist-test-slave-g1xh krb5kdc[25177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:04 dist-test-slave-g1xh krb5kdc[25177](info): set up 2 sockets
Sep 25 02:56:04 dist-test-slave-g1xh krb5kdc[25177](info): commencing operation
krb5kdc: starting...
W20260925 02:56:05.486397 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:56:05 dist-test-slave-g1xh krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:56:05.770644 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:43373
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:43373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:06.105695 25191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:06.105980 25191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:06.106099 25191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:06.113306 25191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:06.113411 25191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:06.113463 25191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:06.113526 25191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:06.113570 25191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:06.123595 25191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:43373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:43373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:06.124787 25191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:06.126046 25191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:06.137558 25199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:06.137576 25197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:06.137830 25196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:06.138790 25191 server_base.cc:1034] running on GCE node
I20260925 02:56:06.139642 25191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:06.141064 25191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:06.142290 25191 hybrid_clock.cc:648] HybridClock initialized: now 1790304966142259 us; error 44 us; skew 500 ppm
I20260925 02:56:06.142640 25191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:06 dist-test-slave-g1xh krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304966, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:06.145793 25191 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:56:06.147245 25191 webserver.cc:460] Webserver started at http://127.2.172.126:45231/ using document root <none> and password file <none>
I20260925 02:56:06.147970 25191 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:06.148101 25191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:06.148413 25191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:06.151381 25191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "4bd8af698c8c43389810478b6a1f7469"
format_stamp: "Formatted at 2026-09-25 02:56:06 on dist-test-slave-g1xh"
server_key: "fcc59ab72528d114315a40922fde25d2"
server_key_iv: "fdd0375b21cffb749ef170037425e72c"
server_key_version: "encryptionkey@0"
I20260925 02:56:06.152218 25191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "4bd8af698c8c43389810478b6a1f7469"
format_stamp: "Formatted at 2026-09-25 02:56:06 on dist-test-slave-g1xh"
server_key: "fcc59ab72528d114315a40922fde25d2"
server_key_iv: "fdd0375b21cffb749ef170037425e72c"
server_key_version: "encryptionkey@0"
I20260925 02:56:06.157311 25191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:56:06.160804 25206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:06.162331 25191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:06.162503 25191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "4bd8af698c8c43389810478b6a1f7469"
format_stamp: "Formatted at 2026-09-25 02:56:06 on dist-test-slave-g1xh"
server_key: "fcc59ab72528d114315a40922fde25d2"
server_key_iv: "fdd0375b21cffb749ef170037425e72c"
server_key_version: "encryptionkey@0"
I20260925 02:56:06.162708 25191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:06.185391 25191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:06.189549 25191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:06.189841 25191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:06.208724 25191 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:43373
I20260925 02:56:06.208756 25257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:43373 every 8 connection(s)
I20260925 02:56:06.210099 25191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:06.213958 25258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:06.218086 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 25191
I20260925 02:56:06.218272 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:56:06.218746 2737 external_mini_cluster.cc:1428] Setting key d6efb09d0f02fb3e1b706ab805f40ff8
I20260925 02:56:06.226140 25258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469: Bootstrap starting.
I20260925 02:56:06.229741 25258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:06.231040 25258 log.cc:824] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:06.234146 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:06.220528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60317 (local address 127.2.172.126:43373)
0925 02:56:06.221411 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:06.221426 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:06.221473 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:56:06.224297 (+ 2824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:06.224325 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:06.224802 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:06.225243 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:06.225742 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:06.227723 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:06.229101 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:06.230237 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:06.230413 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:06.231533 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:06.231568 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:06.231590 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:06.232231 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:06.232340 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:06.232370 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:06.232488 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:06.232676 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 02:56:06.232919 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":377,"threads_started":1}
I20260925 02:56:06.234336 25258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469: No bootstrap required, opened a new log
I20260925 02:56:06.237851 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:06.242712 25258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd8af698c8c43389810478b6a1f7469" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43373 } }
I20260925 02:56:06.243242 25258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:06.243371 25258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd8af698c8c43389810478b6a1f7469, State: Initialized, Role: FOLLOWER
I20260925 02:56:06.244127 25258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd8af698c8c43389810478b6a1f7469" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43373 } }
I20260925 02:56:06.244424 25258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:06.244568 25258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:06.244751 25258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:06.247515 25258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd8af698c8c43389810478b6a1f7469" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43373 } }
I20260925 02:56:06.248172 25258 leader_election.cc:302] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd8af698c8c43389810478b6a1f7469; no voters:
I20260925 02:56:06.248903 25258 leader_election.cc:288] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:06.249050 25268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:06.249135 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:06.242331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55105 (local address 127.2.172.126:43373)
0925 02:56:06.242511 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:06.242521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:06.242901 (+ 380us) server_negotiation.cc:407] Connection header received
0925 02:56:06.243068 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:06.243073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:06.243194 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:06.243321 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:06.243909 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:06.245032 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:06.246323 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:06.247031 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:06.247161 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:06.248253 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:06.248259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:06.248264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:06.248286 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:06.248334 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:06.248355 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:06.248449 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:06.248564 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 02:56:06.248679 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:56:06.251334 25258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:06.251803 25268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [term 1 LEADER]: Becoming Leader. State: Replica: 4bd8af698c8c43389810478b6a1f7469, State: Running, Role: LEADER
I20260925 02:56:06.252324 25268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd8af698c8c43389810478b6a1f7469" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43373 } }
I20260925 02:56:06.255486 25270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bd8af698c8c43389810478b6a1f7469. Latest consensus state: current_term: 1 leader_uuid: "4bd8af698c8c43389810478b6a1f7469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd8af698c8c43389810478b6a1f7469" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43373 } } }
I20260925 02:56:06.255541 25269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bd8af698c8c43389810478b6a1f7469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd8af698c8c43389810478b6a1f7469" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 43373 } } }
I20260925 02:56:06.255914 25270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:06.255967 25269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:06.258119 25277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:06.264366 25277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:06.273183 25277 catalog_manager.cc:1347] Generated new cluster ID: 9fa75dc4c6d848eeb8d593e987a85736
I20260925 02:56:06.273300 25277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:06.291627 25277 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:06.292896 25277 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:06.309110 25277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bd8af698c8c43389810478b6a1f7469: Generated new TSK 0
I20260925 02:56:06.309962 25277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:06.320904 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 25191
W20260925 02:56:06.322144 25253 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60317
W20260925 02:56:06.322908 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 25191: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:06.323058 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 25191
2026-09-25T02:56:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:06 dist-test-slave-g1xh krb5kdc[25289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:06 dist-test-slave-g1xh krb5kdc[25289](info): set up 2 sockets
Sep 25 02:56:06 dist-test-slave-g1xh krb5kdc[25289](info): commencing operation
krb5kdc: starting...
W20260925 02:56:07.438233 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:56:07 dist-test-slave-g1xh krb5kdc[25289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:56:07.723160 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:36731
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:43725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:36731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:08.065529 25302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:08.065879 25302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:08.066045 25302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:08.073477 25302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:08.073645 25302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:08.073750 25302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:08.073856 25302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:08.073930 25302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:08.084323 25302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:43725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:36731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:36731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:08.085775 25302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:08.087108 25302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:08.099846 25307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:08.099794 25308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:08.099750 25310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:08.100472 25302 server_base.cc:1034] running on GCE node
I20260925 02:56:08.101380 25302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:08.102857 25302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:08.104099 25302 hybrid_clock.cc:648] HybridClock initialized: now 1790304968104016 us; error 63 us; skew 500 ppm
I20260925 02:56:08.104447 25302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:08 dist-test-slave-g1xh krb5kdc[25289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304968, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:08.107651 25302 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:56:08.109148 25302 webserver.cc:460] Webserver started at http://127.2.172.126:43899/ using document root <none> and password file <none>
I20260925 02:56:08.109803 25302 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:08.109902 25302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:08.110150 25302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:08.113170 25302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "de46a1558451430f8b7a2568c759c454"
format_stamp: "Formatted at 2026-09-25 02:56:08 on dist-test-slave-g1xh"
server_key: "bfcc7ff456f25f774484e60b023b10aa"
server_key_iv: "6f3d8b951a5206a5e0b0cd4751d8a369"
server_key_version: "encryptionkey@0"
I20260925 02:56:08.113904 25302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "de46a1558451430f8b7a2568c759c454"
format_stamp: "Formatted at 2026-09-25 02:56:08 on dist-test-slave-g1xh"
server_key: "bfcc7ff456f25f774484e60b023b10aa"
server_key_iv: "6f3d8b951a5206a5e0b0cd4751d8a369"
server_key_version: "encryptionkey@0"
I20260925 02:56:08.119081 25302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 02:56:08.122578 25317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:08.123924 25302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:56:08.124114 25302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "de46a1558451430f8b7a2568c759c454"
format_stamp: "Formatted at 2026-09-25 02:56:08 on dist-test-slave-g1xh"
server_key: "bfcc7ff456f25f774484e60b023b10aa"
server_key_iv: "6f3d8b951a5206a5e0b0cd4751d8a369"
server_key_version: "encryptionkey@0"
I20260925 02:56:08.124365 25302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:08.130244 25302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:08.134188 25302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:08.134477 25302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:08.153025 25302 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:36731
I20260925 02:56:08.153059 25368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:36731 every 8 connection(s)
I20260925 02:56:08.154408 25302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:08.158070 25369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:08.160137 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 25302
I20260925 02:56:08.160316 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:56:08.160727 2737 external_mini_cluster.cc:1428] Setting key 95e655de7cd8755d6eaecc2128113a80
I20260925 02:56:08.168941 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:08.162506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40725 (local address 127.2.172.126:36731)
0925 02:56:08.163376 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:08.163397 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:08.163460 (+ 63us) server_negotiation.cc:407] Connection header received
0925 02:56:08.165170 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:08.165199 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:08.165654 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:08.166075 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:08.166979 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:08.166986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:08.167001 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:08.167019 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:08.167098 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:08.167128 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:08.167369 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:08.167489 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 02:56:08.167706 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":355,"threads_started":1}
I20260925 02:56:08.172221 25369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454: Bootstrap starting.
I20260925 02:56:08.172539 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:08.175416 25369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:08.176786 25369 log.cc:824] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:08.179317 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:08.176684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48741 (local address 127.2.172.126:36731)
0925 02:56:08.176838 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:08.176842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:08.177325 (+ 483us) server_negotiation.cc:407] Connection header received
0925 02:56:08.177521 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:08.177530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:08.177585 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:08.177690 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:08.178642 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:08.178648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:08.178652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:08.178670 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:08.178716 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:08.178732 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:08.178847 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:08.178975 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 02:56:08.179038 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:56:08.179968 25369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454: No bootstrap required, opened a new log
I20260925 02:56:08.188354 25369 raft_consensus.cc:348] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de46a1558451430f8b7a2568c759c454" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36731 } }
I20260925 02:56:08.188935 25369 raft_consensus.cc:374] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:08.189061 25369 raft_consensus.cc:729] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de46a1558451430f8b7a2568c759c454, State: Initialized, Role: FOLLOWER
I20260925 02:56:08.189728 25369 consensus_queue.cc:260] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de46a1558451430f8b7a2568c759c454" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36731 } }
I20260925 02:56:08.189944 25369 raft_consensus.cc:388] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:08.190047 25369 raft_consensus.cc:482] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:08.190176 25369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:08.192690 25369 raft_consensus.cc:504] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de46a1558451430f8b7a2568c759c454" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36731 } }
I20260925 02:56:08.193169 25369 leader_election.cc:302] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de46a1558451430f8b7a2568c759c454; no voters:
I20260925 02:56:08.193733 25369 leader_election.cc:288] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:08.193876 25379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:08.194764 25379 raft_consensus.cc:686] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [term 1 LEADER]: Becoming Leader. State: Replica: de46a1558451430f8b7a2568c759c454, State: Running, Role: LEADER
I20260925 02:56:08.195279 25379 consensus_queue.cc:237] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de46a1558451430f8b7a2568c759c454" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36731 } }
I20260925 02:56:08.196012 25369 sys_catalog.cc:564] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:08.198307 25381 sys_catalog.cc:455] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de46a1558451430f8b7a2568c759c454. Latest consensus state: current_term: 1 leader_uuid: "de46a1558451430f8b7a2568c759c454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de46a1558451430f8b7a2568c759c454" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36731 } } }
I20260925 02:56:08.198810 25381 sys_catalog.cc:458] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:08.199460 25380 sys_catalog.cc:455] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de46a1558451430f8b7a2568c759c454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de46a1558451430f8b7a2568c759c454" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 36731 } } }
I20260925 02:56:08.199784 25380 sys_catalog.cc:458] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:08.205016 25385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:08.209928 25385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:08.218312 25385 catalog_manager.cc:1347] Generated new cluster ID: 9fcce0b1c12444b68db1a49583e7d512
I20260925 02:56:08.218438 25385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:08.252144 25385 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:08.252586 25385 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:08.265935 25385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de46a1558451430f8b7a2568c759c454: Generated new TSK 0
I20260925 02:56:08.266853 25385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:08.284029 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 25302
W20260925 02:56:08.285147 25364 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40725
W20260925 02:56:08.285866 2737 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 25302: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:08.286003 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 25302
2026-09-25T02:56:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1963 ms)
[----------] 18 tests from AuthTokenIssuingTest (36061 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:08 dist-test-slave-g1xh krb5kdc[25400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:08 dist-test-slave-g1xh krb5kdc[25400](info): set up 2 sockets
Sep 25 02:56:08 dist-test-slave-g1xh krb5kdc[25400](info): commencing operation
krb5kdc: starting...
W20260925 02:56:09.412374 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:56:09 dist-test-slave-g1xh krb5kdc[25400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:56:09.707710 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:39707
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:36173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:39707,127.2.172.125:33287,127.2.172.124:34655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:10.040859 25413 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:56:10.041208 25413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:10.041339 25413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:10.041529 25413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:10.048905 25413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:10.049000 25413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:10.049062 25413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:10.049105 25413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:10.049217 25413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:10.059194 25413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:36173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:39707,127.2.172.125:33287,127.2.172.124:34655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:39707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:10.060545 25413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:10.061826 25413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:10.073258 25419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:10.073335 25421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:10.073340 25418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:10.074465 25413 server_base.cc:1034] running on GCE node
I20260925 02:56:10.075212 25413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:10.076630 25413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:10.077874 25413 hybrid_clock.cc:648] HybridClock initialized: now 1790304970077801 us; error 47 us; skew 500 ppm
I20260925 02:56:10.078200 25413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:10 dist-test-slave-g1xh krb5kdc[25400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304970, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:10.090970 25413 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:56:10.092738 25413 webserver.cc:460] Webserver started at http://127.2.172.126:45821/ using document root <none> and password file <none>
I20260925 02:56:10.093576 25413 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:10.093722 25413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:10.094077 25413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:10.097359 25413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "87b47ce885194b219613c6b130e321ad"
format_stamp: "Formatted at 2026-09-25 02:56:10 on dist-test-slave-g1xh"
server_key: "7241fe24d1e78a8bba42efe21cf31dfe"
server_key_iv: "48c16271e2c39c6e56129fa8d3305b2b"
server_key_version: "encryptionkey@0"
I20260925 02:56:10.098194 25413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "87b47ce885194b219613c6b130e321ad"
format_stamp: "Formatted at 2026-09-25 02:56:10 on dist-test-slave-g1xh"
server_key: "7241fe24d1e78a8bba42efe21cf31dfe"
server_key_iv: "48c16271e2c39c6e56129fa8d3305b2b"
server_key_version: "encryptionkey@0"
I20260925 02:56:10.103648 25413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 02:56:10.107100 25428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:10.108495 25413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:10.108680 25413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "87b47ce885194b219613c6b130e321ad"
format_stamp: "Formatted at 2026-09-25 02:56:10 on dist-test-slave-g1xh"
server_key: "7241fe24d1e78a8bba42efe21cf31dfe"
server_key_iv: "48c16271e2c39c6e56129fa8d3305b2b"
server_key_version: "encryptionkey@0"
I20260925 02:56:10.108908 25413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:10.142336 25413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:10.146552 25413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:10.146839 25413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:10.165964 25413 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:39707
I20260925 02:56:10.165987 25479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:39707 every 8 connection(s)
I20260925 02:56:10.167351 25413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:10.171289 25480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:10.174825 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 25413
I20260925 02:56:10.175055 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:56:10.175762 2737 external_mini_cluster.cc:1428] Setting key 586bd40efbcda0a19068c5c836d937d4
I20260925 02:56:10.178155 25480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:56:10 dist-test-slave-g1xh krb5kdc[25400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304970, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.172.125@KRBTEST.COM; defaulting to no policy
I20260925 02:56:10.201025 25482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.181513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.2.172.126:39707)
0925 02:56:10.182368 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:10.182388 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:10.182860 (+ 472us) server_negotiation.cc:407] Connection header received
0925 02:56:10.185012 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:10.185018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:10.185258 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:10.185474 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:10.186765 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.188039 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.189435 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.189997 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.190203 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:10.193614 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:10.193623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:10.193636 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:10.193665 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:10.196584 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:10.197296 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:10.197309 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:10.197312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:10.197454 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:10.197832 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:10.197840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:10.197843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:10.198342 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:10.198558 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:10.199080 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 02:56:10.199500 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":597,"thread_start_us":389,"threads_started":1}
I20260925 02:56:10.201004 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.181522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39707 (local address 127.0.0.1:58638)
0925 02:56:10.182689 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.182729 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:10.184620 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:10.185493 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:10.185500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:10.186222 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:10.186551 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:10.186568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.188193 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.188200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:10.189291 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:10.189304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.190234 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.190240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:10.190421 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:10.191096 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:10.191119 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:10.193377 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:10.196724 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:10.196737 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:10.196750 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:10.197166 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:10.197564 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:10.197570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:10.197573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:10.197731 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:10.198492 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:10.198500 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:10.198724 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 02:56:10.198974 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 02:56:10.199649 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":154,"thread_start_us":737,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.125@KRBTEST.COM" created.
I20260925 02:56:10.204260 25480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:56:10.205376 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.204952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54684)
0925 02:56:10.205137 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.205261 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 02:56:10.205617 25429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
I20260925 02:56:10.207283 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.206883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54686)
0925 02:56:10.207035 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.207114 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:56:10.207523 25480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:33287: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.125.keytab.
Entry for principal kudu/127.2.172.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.125.keytab.
I20260925 02:56:10.235143 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.125:33287
--webserver_interface=127.2.172.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.125.keytab
--principal=kudu/127.2.172.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:36173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:39707,127.2.172.125:33287,127.2.172.124:34655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:56:10.256940 25480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } attempt: 1
I20260925 02:56:10.258196 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.257747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54690)
0925 02:56:10.257950 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.258065 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 02:56:10.259253 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.258904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54700)
0925 02:56:10.259055 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.259123 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:56:10.259438 25480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:33287: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
I20260925 02:56:10.327718 25480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } attempt: 2
I20260925 02:56:10.328975 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.328444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54710)
0925 02:56:10.328725 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.328838 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 02:56:10.330009 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.329661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54712)
0925 02:56:10.329820 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.329884 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:56:10.330226 25480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:33287: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
I20260925 02:56:10.421453 25480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } attempt: 3
I20260925 02:56:10.422575 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.422129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54718)
0925 02:56:10.422339 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.422437 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 02:56:10.423621 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.423231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54732)
0925 02:56:10.423376 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.423440 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 02:56:10.423811 25480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:33287: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
I20260925 02:56:10.567050 25480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } attempt: 4
I20260925 02:56:10.568352 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.567776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54734)
0925 02:56:10.568108 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.568208 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260925 02:56:10.569585 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.569050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54738)
0925 02:56:10.569268 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.569405 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260925 02:56:10.569839 25480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:33287: Network error: Client connection negotiation failed: client connection to 127.2.172.125:33287: connect: Connection refused (error 111)
W20260925 02:56:10.577349 25487 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:56:10.577667 25487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:10.577741 25487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:10.577826 25487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:10.585278 25487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:10.585373 25487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:10.585431 25487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:10.585482 25487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:10.585538 25487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:10.595690 25487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:36173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:39707,127.2.172.125:33287,127.2.172.124:34655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.125.keytab
--principal=kudu/127.2.172.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.125:33287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.172.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:10.596967 25487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:10.598251 25487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:10.610522 25492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:10.610550 25495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:10.610574 25493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:10.611457 25487 server_base.cc:1034] running on GCE node
I20260925 02:56:10.612377 25487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:10.613879 25487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:10.615118 25487 hybrid_clock.cc:648] HybridClock initialized: now 1790304970615052 us; error 46 us; skew 500 ppm
I20260925 02:56:10.615463 25487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:10 dist-test-slave-g1xh krb5kdc[25400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304970, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:10.618628 25487 init.cc:377] Logged in from keytab as kudu/127.2.172.125@KRBTEST.COM (short username kudu)
I20260925 02:56:10.620101 25487 webserver.cc:460] Webserver started at http://127.2.172.125:40811/ using document root <none> and password file <none>
I20260925 02:56:10.620818 25487 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:10.620950 25487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:10.621273 25487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:10.624337 25487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/data/instance:
uuid: "ad07828ff8054ec3ac633b81ba44e49d"
format_stamp: "Formatted at 2026-09-25 02:56:10 on dist-test-slave-g1xh"
server_key: "7713b0be8fb114082922a349f9b39b5d"
server_key_iv: "2982a37a01fd2d5f82629a37a448bfc1"
server_key_version: "encryptionkey@0"
I20260925 02:56:10.625110 25487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/wal/instance:
uuid: "ad07828ff8054ec3ac633b81ba44e49d"
format_stamp: "Formatted at 2026-09-25 02:56:10 on dist-test-slave-g1xh"
server_key: "7713b0be8fb114082922a349f9b39b5d"
server_key_iv: "2982a37a01fd2d5f82629a37a448bfc1"
server_key_version: "encryptionkey@0"
I20260925 02:56:10.630476 25487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260925 02:56:10.633875 25503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:10.635196 25487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:56:10.635380 25487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/wal
uuid: "ad07828ff8054ec3ac633b81ba44e49d"
format_stamp: "Formatted at 2026-09-25 02:56:10 on dist-test-slave-g1xh"
server_key: "7713b0be8fb114082922a349f9b39b5d"
server_key_iv: "2982a37a01fd2d5f82629a37a448bfc1"
server_key_version: "encryptionkey@0"
I20260925 02:56:10.635661 25487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:10.648783 25487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:10.652894 25487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:10.653204 25487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:10.672228 25487 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.125:33287
I20260925 02:56:10.672250 25555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.125:33287 every 8 connection(s)
I20260925 02:56:10.673619 25487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
I20260925 02:56:10.677313 25556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:10.682050 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 25487
I20260925 02:56:10.682261 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-1/wal/instance
I20260925 02:56:10.682808 2737 external_mini_cluster.cc:1428] Setting key 5d399a94a59b3e2203088963d399b177
I20260925 02:56:10.685084 25556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:56:10 dist-test-slave-g1xh krb5kdc[25400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304970, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.125@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:56:10.707598 25482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.688477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.2.172.126:39707)
0925 02:56:10.688695 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:10.688701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:10.689982 (+ 1281us) server_negotiation.cc:407] Connection header received
0925 02:56:10.692040 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:10.692048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:10.692149 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:10.692290 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:10.693949 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.694638 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.697158 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.697564 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.697696 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:10.701971 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:10.701976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:10.701980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:10.702012 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:10.704617 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:10.705344 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:10.705348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:10.705350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:10.705467 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:10.705873 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:10.705879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:10.705882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:10.706162 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:10.706330 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:10.707026 (+ 696us) server_negotiation.cc:299] Negotiation successful
0925 02:56:10.707253 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:56:10.708580 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.688978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39707 (local address 127.0.0.1:58644)
0925 02:56:10.689792 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.689845 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:10.691761 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:10.692517 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:10.692525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:10.693074 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:10.693758 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:10.693776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.694804 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.694812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:10.696971 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:10.696989 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.697826 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.697836 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:10.698168 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:10.699013 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:10.699047 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:10.701719 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:10.704765 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:10.704773 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:10.704790 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:10.705211 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:10.705576 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:10.705581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:10.705584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:10.705756 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:10.706295 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:10.706306 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:10.706649 (+ 343us) client_negotiation.cc:769] Sending connection context
0925 02:56:10.706911 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 02:56:10.707412 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":361,"threads_started":1}
I20260925 02:56:10.710234 25556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.172.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.124@KRBTEST.COM" created.
Sep 25 02:56:10 dist-test-slave-g1xh krb5kdc[25400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304970, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.125@KRBTEST.COM for kudu/127.2.172.125@KRBTEST.COM
I20260925 02:56:10.723883 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.710981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54748)
0925 02:56:10.711137 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.711156 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:10.711240 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:10.712131 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:10.712137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:10.712566 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:10.712735 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:10.712747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.714091 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.714097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:10.714907 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:10.714917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.715748 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.715753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:10.715973 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:10.716510 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:10.716527 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:10.718428 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:10.721461 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:10.721466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:10.721471 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:10.721822 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:10.722195 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:10.722200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:10.722203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:10.722338 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:10.722908 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:10.722913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:10.723054 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 02:56:10.723190 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 02:56:10.723452 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":3}
I20260925 02:56:10.723940 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.711118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.2.172.125:33287)
0925 02:56:10.711722 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:10.711727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:10.711771 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:56:10.711826 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:10.711832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:10.711998 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:10.712143 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:10.712903 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.713958 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.715042 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.715619 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.715751 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:10.718636 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:10.718647 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:10.718659 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:10.718685 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:10.721322 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:10.721962 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:10.721971 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:10.721974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:10.722089 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:10.722451 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:10.722456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:10.722459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:10.722773 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:10.722973 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:10.723331 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 02:56:10.723577 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":462,"thread_start_us":285,"threads_started":1}
I20260925 02:56:10.726712 25556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:56:10.727993 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.727393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:59954)
0925 02:56:10.727637 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.727783 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 02:56:10.728317 25505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
I20260925 02:56:10.729956 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.729526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:59958)
0925 02:56:10.729704 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.729807 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 02:56:10.730171 25556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:34655: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.124.keytab.
Entry for principal kudu/127.2.172.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.124.keytab.
I20260925 02:56:10.746389 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.124:34655
--webserver_interface=127.2.172.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.124.keytab
--principal=kudu/127.2.172.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:36173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:39707,127.2.172.125:33287,127.2.172.124:34655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:56:10.779505 25556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } attempt: 1
I20260925 02:56:10.780786 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.780241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:59970)
0925 02:56:10.780454 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.780595 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 02:56:10.781824 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.781455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:59982)
0925 02:56:10.781618 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.781704 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:56:10.782071 25556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:34655: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
I20260925 02:56:10.850328 25556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } attempt: 2
I20260925 02:56:10.851485 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.851053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:59984)
0925 02:56:10.851242 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.851332 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 02:56:10.852473 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.852156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:59992)
0925 02:56:10.852290 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.852362 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:56:10.852708 25556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:34655: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
I20260925 02:56:10.869088 25480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } attempt: 5
Sep 25 02:56:10 dist-test-slave-g1xh krb5kdc[25400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304970, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for kudu/127.2.172.125@KRBTEST.COM
I20260925 02:56:10.884333 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.869825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54752)
0925 02:56:10.870031 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.870052 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:10.870189 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:10.870548 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:10.870553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:10.871083 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:10.871369 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:10.871381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.872381 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.872386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:10.873999 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:10.874019 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.874777 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.874786 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:10.875058 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:10.875816 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:10.875846 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:10.878261 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:10.881917 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:10.881921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:10.881924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:10.882305 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:10.882733 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:10.882737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:10.882739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:10.882863 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:10.883543 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:10.883583 (+ 40us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:10.883740 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 02:56:10.883867 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 02:56:10.884078 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 02:56:10.884542 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.869926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.2.172.125:33287)
0925 02:56:10.870112 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:10.870117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:10.870135 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:56:10.870329 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:10.870333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:10.870392 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:10.870475 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:10.871512 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.872247 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.874165 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:10.874607 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:10.874704 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:10.878472 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:10.878479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:10.878484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:10.878525 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:10.881769 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:10.882424 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:10.882433 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:10.882438 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:10.882607 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:10.882963 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:10.882968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:10.882971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:10.883418 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:10.883635 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:10.884007 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 02:56:10.884241 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:56:10.885552 25480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:56:10.886783 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.886426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60008)
0925 02:56:10.886587 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.886660 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 02:56:10.887722 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.887350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60022)
0925 02:56:10.887509 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.887628 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 02:56:10.887916 25480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:34655: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
I20260925 02:56:10.939180 25480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } attempt: 1
I20260925 02:56:10.940469 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.940015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60028)
0925 02:56:10.940206 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.940296 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 02:56:10.941395 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.941070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60032)
0925 02:56:10.941228 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.941291 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:56:10.941629 25480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:34655: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
I20260925 02:56:10.943907 25556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } attempt: 3
I20260925 02:56:10.944798 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.944398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60046)
0925 02:56:10.944565 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.944667 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 02:56:10.945758 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:10.945363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60058)
0925 02:56:10.945571 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:10.945650 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260925 02:56:10.945995 25556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:34655: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
I20260925 02:56:11.009881 25480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } attempt: 2
I20260925 02:56:11.011117 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.010602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60068)
0925 02:56:11.010811 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:11.010933 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 02:56:11.012115 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.011770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60072)
0925 02:56:11.011910 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:11.012005 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:56:11.012315 25480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:34655: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
I20260925 02:56:11.089212 25556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } attempt: 4
I20260925 02:56:11.090291 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.089820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60080)
0925 02:56:11.090047 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:11.090131 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 02:56:11.090309 25563 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:56:11.090735 25563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:11.090853 25563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:11.091022 25563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 02:56:11.091466 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.090986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60086)
0925 02:56:11.091239 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:11.091330 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260925 02:56:11.091707 25556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:34655: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
W20260925 02:56:11.098690 25563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:11.098784 25563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:11.098842 25563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:11.098893 25563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:11.098955 25563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:11.109112 25563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:36173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:39707,127.2.172.125:33287,127.2.172.124:34655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.124.keytab
--principal=kudu/127.2.172.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.124:34655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.172.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:11.110435 25563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:11.111785 25563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:56:11.118516 25480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } attempt: 3
I20260925 02:56:11.119588 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.119107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60092)
0925 02:56:11.119321 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:11.119400 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 02:56:11.120536 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.120186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60108)
0925 02:56:11.120365 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:11.120437 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260925 02:56:11.120805 25480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:34655: Network error: Client connection negotiation failed: client connection to 127.2.172.124:34655: connect: Connection refused (error 111)
W20260925 02:56:11.128255 25571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:11.128185 25568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:11.128221 25569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:11.128791 25563 server_base.cc:1034] running on GCE node
I20260925 02:56:11.129647 25563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:11.131181 25563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:11.132433 25563 hybrid_clock.cc:648] HybridClock initialized: now 1790304971132361 us; error 44 us; skew 500 ppm
I20260925 02:56:11.132771 25563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:11 dist-test-slave-g1xh krb5kdc[25400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304971, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:11.135874 25563 init.cc:377] Logged in from keytab as kudu/127.2.172.124@KRBTEST.COM (short username kudu)
I20260925 02:56:11.137265 25563 webserver.cc:460] Webserver started at http://127.2.172.124:43459/ using document root <none> and password file <none>
I20260925 02:56:11.137941 25563 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:11.138069 25563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:11.138398 25563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:11.141563 25563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/data/instance:
uuid: "d070f608467844de83f63d8c027fd72b"
format_stamp: "Formatted at 2026-09-25 02:56:11 on dist-test-slave-g1xh"
server_key: "4a016fa8f766886281dceb9438341ac2"
server_key_iv: "7031e7a38c78c9ecadb029eb44228429"
server_key_version: "encryptionkey@0"
I20260925 02:56:11.142445 25563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/wal/instance:
uuid: "d070f608467844de83f63d8c027fd72b"
format_stamp: "Formatted at 2026-09-25 02:56:11 on dist-test-slave-g1xh"
server_key: "4a016fa8f766886281dceb9438341ac2"
server_key_iv: "7031e7a38c78c9ecadb029eb44228429"
server_key_version: "encryptionkey@0"
I20260925 02:56:11.147524 25563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:56:11.150897 25578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:11.152268 25563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 02:56:11.152458 25563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/wal
uuid: "d070f608467844de83f63d8c027fd72b"
format_stamp: "Formatted at 2026-09-25 02:56:11 on dist-test-slave-g1xh"
server_key: "4a016fa8f766886281dceb9438341ac2"
server_key_iv: "7031e7a38c78c9ecadb029eb44228429"
server_key_version: "encryptionkey@0"
I20260925 02:56:11.152717 25563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:11.179291 25563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:11.184080 25563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:11.184386 25563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:11.202390 25563 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.124:34655
I20260925 02:56:11.202445 25631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.124:34655 every 8 connection(s)
I20260925 02:56:11.203833 25563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
I20260925 02:56:11.207659 25632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:11.213373 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 25563
I20260925 02:56:11.213590 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304596269475-2737-0/minicluster-data/master-2/wal/instance
I20260925 02:56:11.214200 2737 external_mini_cluster.cc:1428] Setting key 602b4582dd4ca248abf6c1be121e30e8
I20260925 02:56:11.214916 25632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:56:11 dist-test-slave-g1xh krb5kdc[25400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304971, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.124@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
Sep 25 02:56:11 dist-test-slave-g1xh krb5kdc[25400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.124@KRBTEST.COM
I20260925 02:56:11.236778 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.217995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.2.172.126:39707)
0925 02:56:11.218537 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.218545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.219123 (+ 578us) server_negotiation.cc:407] Connection header received
0925 02:56:11.219608 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.219613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.219689 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.219819 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.221603 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.222816 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.224956 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.225610 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.225744 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.230287 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:11.230295 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:11.230301 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:11.230347 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:11.233151 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.234126 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.234132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.234134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.234263 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.234725 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.234732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.234736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.235097 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:11.235276 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:11.236161 (+ 885us) server_negotiation.cc:299] Negotiation successful
0925 02:56:11.236345 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":256,"threads_started":1}
I20260925 02:56:11.238034 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.218100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:39707 (local address 127.0.0.1:58660)
0925 02:56:11.218960 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:11.219008 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:11.219362 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:11.219903 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:11.219910 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:11.220857 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:11.221415 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:11.221445 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.222982 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.222986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:11.224801 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:11.224819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.225838 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.225843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:11.226034 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.226950 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:11.226981 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:11.230043 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:11.233298 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:11.233309 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:11.233329 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:11.233987 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:11.234374 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:11.234383 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:11.234387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:11.234611 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:11.235238 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:11.235244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:11.235741 (+ 497us) client_negotiation.cc:769] Sending connection context
0925 02:56:11.236093 (+ 352us) client_negotiation.cc:240] Negotiation successful
0925 02:56:11.236596 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":83,"thread_start_us":438,"threads_started":1}
I20260925 02:56:11.238054 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.216316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.2.172.124:34655)
0925 02:56:11.217049 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.217060 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.217101 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:56:11.219354 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.219377 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.219693 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.219885 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.221722 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.223352 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.225077 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.225935 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.226028 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.230964 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:11.230980 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:11.230998 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:11.231039 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:11.234503 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.235344 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.235352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.235355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.235507 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.235918 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.235923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.235925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.236344 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:11.236638 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:11.237070 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 02:56:11.237313 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":533,"spinlock_wait_cycles":26496,"thread_start_us":321,"threads_started":1}
I20260925 02:56:11.239806 25632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:56:11 dist-test-slave-g1xh krb5kdc[25400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304971, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.124@KRBTEST.COM for kudu/127.2.172.125@KRBTEST.COM
Sep 25 02:56:11 dist-test-slave-g1xh krb5kdc[25400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:56:11.253279 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.240559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:33287 (local address 127.0.0.1:54762)
0925 02:56:11.240733 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:11.240753 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:11.240865 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:11.241194 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:11.241197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:11.241511 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:11.241662 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:11.241670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.242918 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.242924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:11.243899 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:11.243911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.244627 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.244632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:11.244841 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.245421 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:11.245438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:11.247512 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:11.250828 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:11.250832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:11.250834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:11.251298 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:11.251743 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:11.251746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:11.251749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:11.251892 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:11.252561 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:11.252576 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:11.252708 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 02:56:11.252822 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:56:11.253032 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 02:56:11.253607 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.240609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.2.172.125:33287)
0925 02:56:11.240780 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.240784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.240801 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:56:11.240986 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.240992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.241067 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.241152 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.241838 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.242783 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.244031 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.244476 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.244605 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.247742 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:11.247748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:11.247752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:11.247787 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:11.250692 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.251432 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.251440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.251445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.251630 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.252003 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.252009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.252014 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.252436 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:11.252596 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:11.252971 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 02:56:11.253219 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:56:11.254582 25632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:56:11.256886 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.242011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.2.172.126:39707)
0925 02:56:11.242201 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.242206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.242224 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:56:11.242275 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.242280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.242361 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.242452 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.243616 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.244293 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.245706 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.246130 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.246267 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.250536 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:11.250545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:11.250551 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:11.250596 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:11.254050 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.254742 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.254747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.254750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.254894 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.255281 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.255286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.255289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.255703 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:11.255980 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:11.256287 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 02:56:11.256545 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 02:56:11 dist-test-slave-g1xh krb5kdc[25400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304971, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.124@KRBTEST.COM for kudu/127.2.172.124@KRBTEST.COM
Sep 25 02:56:11 dist-test-slave-g1xh krb5kdc[25400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.125@KRBTEST.COM
I20260925 02:56:11.269255 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.255483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.2.172.124:34655)
0925 02:56:11.255698 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.255704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.255779 (+ 75us) server_negotiation.cc:407] Connection header received
0925 02:56:11.255929 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.255934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.255989 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.256082 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.256813 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.257576 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.258953 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.259356 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.259483 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.262929 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:11.262934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:11.262937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:11.262965 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:11.266382 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.267147 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.267156 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.267162 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.267306 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.267791 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.267794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.267796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.268115 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:11.268244 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:11.268724 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 02:56:11.268896 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 02:56:11.269443 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.255509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60122)
0925 02:56:11.255696 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:11.255715 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:11.255821 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:11.256104 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:11.256108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:11.256472 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:11.256701 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:11.256713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.257698 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.257706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:11.258828 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:11.258841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.259501 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.259506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:11.259732 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.260334 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:11.260352 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:11.262706 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:11.266576 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:11.266581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:11.266584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:11.267003 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:11.267439 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:11.267444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:11.267448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:11.267680 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:11.268250 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:11.268256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:11.268453 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 02:56:11.268634 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 02:56:11.269112 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":105}
I20260925 02:56:11.275388 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.258525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54768 (local address 127.2.172.125:33287)
0925 02:56:11.258676 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.258680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.258695 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:56:11.258830 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.258833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.258888 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.258986 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.260405 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.261104 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.262859 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.263614 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.263753 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.269115 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:11.269121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:11.269125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:11.269167 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:11.272784 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.273448 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.273454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.273458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.273623 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.273991 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.273996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.273999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.274361 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:11.274646 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:11.274825 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 02:56:11.275053 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:56:11.275763 25632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b: Bootstrap starting.
I20260925 02:56:11.279829 25632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:11.280524 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:11.281087 25632 log.cc:824] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:11.284294 25632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b: No bootstrap required, opened a new log
I20260925 02:56:11.283224 25455 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "87b47ce885194b219613c6b130e321ad"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58676
I20260925 02:56:11.292521 25632 raft_consensus.cc:348] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } }
I20260925 02:56:11.292906 25632 raft_consensus.cc:729] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d070f608467844de83f63d8c027fd72b, State: Initialized, Role: FOLLOWER
I20260925 02:56:11.293373 25632 consensus_queue.cc:260] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } }
I20260925 02:56:11.294306 25639 sys_catalog.cc:455] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } } }
I20260925 02:56:11.294641 25639 sys_catalog.cc:458] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:11.295634 25632 sys_catalog.cc:564] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:11.298019 25480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } attempt: 4
Sep 25 02:56:11 dist-test-slave-g1xh krb5kdc[25400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304970, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for kudu/127.2.172.124@KRBTEST.COM
W20260925 02:56:11.310670 25650 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:56:11.310809 25650 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:56:11.314433 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.298761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60138)
0925 02:56:11.299077 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:11.299100 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:11.299243 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:11.299727 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:11.299735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:11.300272 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:11.300482 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:11.300497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.301778 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.301784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:11.303210 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:11.303227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.304097 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.304103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:11.304339 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.304937 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:11.304955 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:11.307035 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:11.311482 (+ 4447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:11.311490 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:11.311494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:11.312040 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:11.312452 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:11.312458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:11.312461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:11.312630 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:11.313348 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:11.313353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:11.313573 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 02:56:11.313735 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 02:56:11.314023 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260925 02:56:11.314486 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.298800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60138 (local address 127.2.172.124:34655)
0925 02:56:11.299041 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.299051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.299182 (+ 131us) server_negotiation.cc:407] Connection header received
0925 02:56:11.299402 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.299410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.299516 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.299696 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.300677 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.301625 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.303352 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.303935 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.304076 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.307214 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:11.307224 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:11.307227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:11.307256 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:11.311302 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.312157 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.312163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.312167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.312315 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.312741 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.312747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.312750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.313220 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:11.313371 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:11.313837 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 02:56:11.314087 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:56:11.303278 25563 master_runner.cc:186] Error getting master registration for 127.2.172.126:39707: OK, instance_id { permanent_uuid: "87b47ce885194b219613c6b130e321ad" instance_seqno: 1790304970160527 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:56:11.317659 25563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:56:11.320861 25480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad: Bootstrap starting.
I20260925 02:56:11.324064 25480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:11.325156 25480 log.cc:824] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:11.327898 25480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad: No bootstrap required, opened a new log
I20260925 02:56:11.333245 25480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } }
I20260925 02:56:11.333597 25480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b47ce885194b219613c6b130e321ad, State: Initialized, Role: FOLLOWER
I20260925 02:56:11.334062 25480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } }
I20260925 02:56:11.335078 25653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } } }
I20260925 02:56:11.335474 25653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:11.336254 25480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:11.336418 25455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 02:56:11.336691 25455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:11.340920 25455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } }
I20260925 02:56:11.343015 25455 leader_election.cc:288] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [CANDIDATE]: Term 1 election: Requested vote from peers ad07828ff8054ec3ac633b81ba44e49d (127.2.172.125:33287), d070f608467844de83f63d8c027fd72b (127.2.172.124:34655)
I20260925 02:56:11.343817 25607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b47ce885194b219613c6b130e321ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d070f608467844de83f63d8c027fd72b"
I20260925 02:56:11.344447 25607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:11.349586 25607 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b47ce885194b219613c6b130e321ad in term 1.
I20260925 02:56:11.350409 25430 leader_election.cc:302] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b47ce885194b219613c6b130e321ad, d070f608467844de83f63d8c027fd72b; no voters:
I20260925 02:56:11.351917 25653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:11.371219 25653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [term 1 LEADER]: Becoming Leader. State: Replica: 87b47ce885194b219613c6b130e321ad, State: Running, Role: LEADER
I20260925 02:56:11.372861 25653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } }
I20260925 02:56:11.390897 25556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } attempt: 5
W20260925 02:56:11.399874 25434 tablet.cc:2369] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:11.407171 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.378241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.2.172.125:33287)
0925 02:56:11.378525 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.378535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.384607 (+ 6072us) server_negotiation.cc:407] Connection header received
0925 02:56:11.385167 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.385176 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.385285 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.385370 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.388224 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.389594 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.393319 (+ 3725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.394205 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.394328 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.398403 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:11.398412 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:11.398418 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:11.398477 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:11.403177 (+ 4700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.404198 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.404204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.404207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.404387 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.405027 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.405035 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.405041 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.405474 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:11.405743 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:11.406375 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 02:56:11.406700 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260925 02:56:11.426784 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.377310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.2.172.126:39707)
0925 02:56:11.378022 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.378032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.408361 (+ 30329us) server_negotiation.cc:407] Connection header received
0925 02:56:11.408672 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.408682 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.408785 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.408863 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.410717 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.411967 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.414260 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.415084 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.415203 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.419201 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:11.419211 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:11.419218 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:11.419269 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:11.423245 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.424226 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.424234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.424240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.424423 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.424908 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.424914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.424920 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.425303 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:11.425590 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:11.426134 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 02:56:11.426350 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511}
W20260925 02:56:11.436440 25585 tablet.cc:2369] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:56:11 dist-test-slave-g1xh krb5kdc[25400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304970, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.125@KRBTEST.COM for kudu/127.2.172.124@KRBTEST.COM
I20260925 02:56:11.435389 25664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b47ce885194b219613c6b130e321ad. Latest consensus state: current_term: 1 leader_uuid: "87b47ce885194b219613c6b130e321ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } } }
I20260925 02:56:11.450419 25664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:11.454074 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.379131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.2.172.124:34655)
0925 02:56:11.379373 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.379381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.420837 (+ 41456us) server_negotiation.cc:407] Connection header received
0925 02:56:11.421249 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.421255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.421380 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.421465 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.431327 (+ 9862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.432417 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.438606 (+ 6189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.439320 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.439431 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.445231 (+ 5800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:11.445239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:11.445245 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:11.445287 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:11.449166 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.450952 (+ 1786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.450958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.450963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.451124 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.451856 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.451864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.451873 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.452224 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:11.452505 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:11.453389 (+ 884us) server_negotiation.cc:299] Negotiation successful
0925 02:56:11.453665 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260925 02:56:11.469264 25674 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:56:11.469401 25674 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:56:11.469527 25676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:11.472842 25676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:11.475035 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.391581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:34655 (local address 127.0.0.1:60150)
0925 02:56:11.391858 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:11.391881 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:11.392002 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:11.425680 (+ 33678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:11.425686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:11.426199 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:11.426406 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:11.426418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.429491 (+ 3073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.429496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:11.430908 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:11.430926 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.441021 (+ 10095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.441030 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:11.441347 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.446132 (+ 4785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:11.446169 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:11.451096 (+ 4927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:11.468207 (+ 17111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:11.468213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:11.468217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:11.468839 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:11.469384 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:11.469392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:11.469397 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:11.469580 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:11.470913 (+ 1333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:11.470918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:11.471147 (+ 229us) client_negotiation.cc:769] Sending connection context
0925 02:56:11.471306 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 02:56:11.474611 (+ 3305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260925 02:56:11.476348 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:11.391686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.2.172.124:34655)
0925 02:56:11.425139 (+ 33453us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.425145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.425168 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:56:11.425267 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.425274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.425407 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.426610 (+ 1203us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.427092 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.429272 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.431213 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:11.434655 (+ 3442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:11.436286 (+ 1631us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:11.451479 (+ 15193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:11.451490 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:11.451498 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:11.451567 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:11.462992 (+ 11425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.469029 (+ 6037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.469034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.469037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.469230 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:11.469721 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:11.469726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:11.469730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:11.470128 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:11.475466 (+ 5338us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:11.475658 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 02:56:11.475926 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33185,"thread_start_us":37245,"threads_started":1}
I20260925 02:56:11.441939 25413 master_runner.cc:186] Error getting master registration for 127.2.172.125:33287: OK, instance_id { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" instance_seqno: 1790304970666713 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:56:11.484581 25413 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:56:11.486915 25556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d: Bootstrap starting.
I20260925 02:56:11.498257 25556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:11.500211 25556 log.cc:824] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:11.510562 25607 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [term 1 FOLLOWER]: Refusing update from remote peer 87b47ce885194b219613c6b130e321ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:11.511776 25664 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:11.524005 25556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d: No bootstrap required, opened a new log
I20260925 02:56:11.539803 25556 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } }
I20260925 02:56:11.540557 25556 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad07828ff8054ec3ac633b81ba44e49d, State: Initialized, Role: FOLLOWER
I20260925 02:56:11.540388 25639 sys_catalog.cc:455] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b47ce885194b219613c6b130e321ad. Latest consensus state: current_term: 1 leader_uuid: "87b47ce885194b219613c6b130e321ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } } }
I20260925 02:56:11.540728 25639 sys_catalog.cc:458] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:11.541241 25556 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } }
I20260925 02:56:11.548655 25682 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } } }
I20260925 02:56:11.551445 25556 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:11.551878 25530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:11.552196 25531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b47ce885194b219613c6b130e321ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ad07828ff8054ec3ac633b81ba44e49d"
I20260925 02:56:11.561407 25530 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [term 1 FOLLOWER]: Refusing update from remote peer 87b47ce885194b219613c6b130e321ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:11.565523 25531 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b47ce885194b219613c6b130e321ad in term 1.
I20260925 02:56:11.566489 25653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87b47ce885194b219613c6b130e321ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } } }
I20260925 02:56:11.567366 25653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:11.567008 25664 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:11.570103 25639 sys_catalog.cc:455] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87b47ce885194b219613c6b130e321ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } } }
I20260925 02:56:11.570374 25639 sys_catalog.cc:458] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:11.565724 25682 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:11.578316 25676 catalog_manager.cc:1347] Generated new cluster ID: e5913493d8b9434395e346fb13d8c0ed
I20260925 02:56:11.578477 25676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:11.608160 25676 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:11.610164 25676 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:11.612272 25688 mvcc.cc:205] Tried to move back new op lower bound from 7333089163279523840 to 7333089162877890560. Current Snapshot: MvccSnapshot[applied={T|T < 7333089163279523840}]
I20260925 02:56:11.621021 25686 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87b47ce885194b219613c6b130e321ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } } }
I20260925 02:56:11.621471 25686 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:11.621294 25664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87b47ce885194b219613c6b130e321ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } } }
I20260925 02:56:11.621654 25664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:11.622008 25682 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b47ce885194b219613c6b130e321ad. Latest consensus state: current_term: 1 leader_uuid: "87b47ce885194b219613c6b130e321ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b47ce885194b219613c6b130e321ad" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 39707 } } peers { permanent_uuid: "ad07828ff8054ec3ac633b81ba44e49d" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 33287 } } peers { permanent_uuid: "d070f608467844de83f63d8c027fd72b" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 34655 } } }
I20260925 02:56:11.622314 25682 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:11.636740 25701 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:56:11.653378 25676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b47ce885194b219613c6b130e321ad: Generated new TSK 0
I20260925 02:56:11.654676 25676 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:56:11.766426 25560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:11.750473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.2.172.125:33287)
0925 02:56:11.750749 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.750758 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.764225 (+ 13467us) server_negotiation.cc:407] Connection header received
0925 02:56:11.764606 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.764615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.764739 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.764825 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.766162 (+ 1337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54780: BlockingRecv error: recv got EOF from 127.0.0.1:54780 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260925 02:56:11.766971 25671 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:11.750041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.2.172.124:34655)
0925 02:56:11.750238 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.750246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.764776 (+ 14530us) server_negotiation.cc:407] Connection header received
0925 02:56:11.765303 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.765311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.765424 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.765509 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.766704 (+ 1195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60162: BlockingRecv error: recv got EOF from 127.0.0.1:60162 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 02:56:11.767706 25636 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:11.750083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.2.172.126:39707)
0925 02:56:11.750291 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.750296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.765673 (+ 15377us) server_negotiation.cc:407] Connection header received
0925 02:56:11.766119 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.766125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.766222 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.766299 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.767465 (+ 1166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58690: BlockingRecv error: recv got EOF from 127.0.0.1:58690 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 02:56:11.775290 25560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:11.773202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.2.172.125:33287)
0925 02:56:11.773419 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.773427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.773909 (+ 482us) server_negotiation.cc:407] Connection header received
0925 02:56:11.774149 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.774153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.774243 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.774350 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.775031 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54786: BlockingRecv error: recv got EOF from 127.0.0.1:54786 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 02:56:11.781790 25560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:11.779812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.2.172.125:33287)
0925 02:56:11.780024 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.780031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.780487 (+ 456us) server_negotiation.cc:407] Connection header received
0925 02:56:11.780680 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.780686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.780765 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.780865 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.781590 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54802: BlockingRecv error: recv got EOF from 127.0.0.1:54802 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 02:56:11.788800 25560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:11.786814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.2.172.125:33287)
0925 02:56:11.787061 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.787069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.787530 (+ 461us) server_negotiation.cc:407] Connection header received
0925 02:56:11.787756 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.787762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.787847 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.787948 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.788612 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54818: BlockingRecv error: recv got EOF from 127.0.0.1:54818 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 02:56:11.799129 25560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:11.796851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.2.172.125:33287)
0925 02:56:11.797224 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.797232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.797781 (+ 549us) server_negotiation.cc:407] Connection header received
0925 02:56:11.798006 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.798012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.798113 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.798206 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.798928 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54824: BlockingRecv error: recv got EOF from 127.0.0.1:54824 (error 108)
Metrics: {"server-negotiator.queue_time_us":185}
W20260925 02:56:11.813592 25560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:11.811237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.2.172.125:33287)
0925 02:56:11.811493 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.811503 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.812105 (+ 602us) server_negotiation.cc:407] Connection header received
0925 02:56:11.812330 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.812337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.812425 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.812532 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.813355 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54838: BlockingRecv error: recv got EOF from 127.0.0.1:54838 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 02:56:11.835809 25560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:11.833324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54854 (local address 127.2.172.125:33287)
0925 02:56:11.833555 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.833564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.834124 (+ 560us) server_negotiation.cc:407] Connection header received
0925 02:56:11.834397 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.834401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.834485 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.834602 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.835517 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54854: BlockingRecv error: recv got EOF from 127.0.0.1:54854 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 02:56:11.874300 25560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:11.871638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.2.172.125:33287)
0925 02:56:11.871890 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.871899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.872468 (+ 569us) server_negotiation.cc:407] Connection header received
0925 02:56:11.872720 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.872727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.872832 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.872935 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.874059 (+ 1124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54858: BlockingRecv error: recv got EOF from 127.0.0.1:54858 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260925 02:56:11.944983 25560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:11.942617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.2.172.125:33287)
0925 02:56:11.942869 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:11.942876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:11.943286 (+ 410us) server_negotiation.cc:407] Connection header received
0925 02:56:11.943504 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:11.943509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:11.943659 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:11.943752 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:11.944748 (+ 996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54874: BlockingRecv error: recv got EOF from 127.0.0.1:54874 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 02:56:12.087162 25560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:12.085325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.2.172.125:33287)
0925 02:56:12.085580 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:12.085587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:12.085846 (+ 259us) server_negotiation.cc:407] Connection header received
0925 02:56:12.086004 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:12.086008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:12.086089 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:12.086181 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:12.086923 (+ 742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54884: BlockingRecv error: recv got EOF from 127.0.0.1:54884 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:56:12.312968 25650 catalog_manager.cc:1259] Loaded cluster ID: e5913493d8b9434395e346fb13d8c0ed
I20260925 02:56:12.313073 25650 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b: loading cluster ID for follower catalog manager: success
I20260925 02:56:12.315496 25650 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b: acquiring CA information for follower catalog manager: success
I20260925 02:56:12.317227 25650 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d070f608467844de83f63d8c027fd72b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 02:56:12.348810 25560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:12.346928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.2.172.125:33287)
0925 02:56:12.347176 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:12.347184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:12.347370 (+ 186us) server_negotiation.cc:407] Connection header received
0925 02:56:12.347623 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:12.347632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:12.347720 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:12.347838 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:12.348588 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50626: BlockingRecv error: recv got EOF from 127.0.0.1:50626 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:56:12.656541 25701 catalog_manager.cc:1259] Loaded cluster ID: e5913493d8b9434395e346fb13d8c0ed
I20260925 02:56:12.656677 25701 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d: loading cluster ID for follower catalog manager: success
I20260925 02:56:12.660522 25701 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d: acquiring CA information for follower catalog manager: success
I20260925 02:56:12.663017 25701 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad07828ff8054ec3ac633b81ba44e49d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:56:12.871701 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:12.864627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.2.172.125:33287)
0925 02:56:12.865489 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:12.865494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:12.865512 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:56:12.865577 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:12.865581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:12.865740 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:12.865845 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:12.866377 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:12.867257 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:12.869209 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:12.869662 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:12.869757 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:12.870338 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:12.870920 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:12.871029 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:12.871196 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 02:56:12.871265 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":576,"threads_started":1}
I20260925 02:56:12.885361 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:12.878699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.2.172.124:34655)
0925 02:56:12.879320 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:12.879324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:12.879341 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:56:12.879436 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:12.879440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:12.879664 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:12.879805 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:12.880280 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:12.880978 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:12.883010 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:12.883487 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:12.883675 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:12.884239 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:12.884733 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:12.884895 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:12.884979 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:56:12.885063 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":407,"threads_started":1}
I20260925 02:56:12.912604 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:12.893776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.2.172.125:33287)
0925 02:56:12.894195 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:12.894201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:12.904991 (+ 10790us) server_negotiation.cc:407] Connection header received
0925 02:56:12.905569 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:12.905576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:12.905682 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:12.905757 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:12.906576 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:12.907808 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:12.910022 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:12.910733 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:12.910836 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:12.911633 (+ 797us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:12.911773 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:12.911850 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:12.912165 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 02:56:12.912257 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260925 02:56:12.914482 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:12.893979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35742 (local address 127.2.172.124:34655)
0925 02:56:12.899283 (+ 5304us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:12.899291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:12.906002 (+ 6711us) server_negotiation.cc:407] Connection header received
0925 02:56:12.906619 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:12.906626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:12.906750 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:12.906853 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:12.907826 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:12.909414 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:12.911794 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:12.912466 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:12.912559 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:12.913225 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:12.913357 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:12.913426 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:12.914150 (+ 724us) server_negotiation.cc:299] Negotiation successful
0925 02:56:12.914235 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5156}
I20260925 02:56:12.918527 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:12.893219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.2.172.126:39707)
0925 02:56:12.911148 (+ 17929us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:12.911155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:12.911178 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:56:12.911291 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:12.911297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:12.911501 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:12.911724 (+ 223us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:12.912526 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:12.913654 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:12.916065 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:12.916696 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:12.916837 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:12.917387 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:12.917838 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:12.917968 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:12.918109 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 02:56:12.918197 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17805,"thread_start_us":19154,"threads_started":1}
I20260925 02:56:12.924175 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 25413
I20260925 02:56:12.925207 25475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:13.036248 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 25413
I20260925 02:56:13.057760 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 25487
I20260925 02:56:13.058874 25551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:13.165266 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 25487
I20260925 02:56:13.186762 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 25563
I20260925 02:56:13.187968 25627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:13.294554 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 25563
2026-09-25T02:56:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5011 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:13 dist-test-slave-g1xh krb5kdc[25791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:13 dist-test-slave-g1xh krb5kdc[25791](info): set up 2 sockets
Sep 25 02:56:13 dist-test-slave-g1xh krb5kdc[25791](info): commencing operation
krb5kdc: starting...
W20260925 02:56:14.434615 2737 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/test-user.keytab.
Sep 25 02:56:14 dist-test-slave-g1xh krb5kdc[25791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
Entry for principal kudu/127.2.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab.
I20260925 02:56:14.722414 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:42873
--webserver_interface=127.2.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:42873,127.2.172.125:39403,127.2.172.124:46825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:15.054394 25804 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:56:15.054751 25804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:15.054893 25804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:15.055020 25804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:15.062366 25804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:15.062453 25804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:15.062564 25804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:15.062652 25804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:15.062742 25804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:15.072638 25804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:42873,127.2.172.125:39403,127.2.172.124:46825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.126.keytab
--principal=kudu/127.2.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.126:42873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:15.073896 25804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:15.075174 25804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:15.087270 25809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:15.087358 25810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:15.087309 25812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:15.088349 25804 server_base.cc:1034] running on GCE node
I20260925 02:56:15.089198 25804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:15.090595 25804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:15.091852 25804 hybrid_clock.cc:648] HybridClock initialized: now 1790304975091768 us; error 49 us; skew 500 ppm
I20260925 02:56:15.092262 25804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:15 dist-test-slave-g1xh krb5kdc[25791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304975, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:15.101415 25804 init.cc:377] Logged in from keytab as kudu/127.2.172.126@KRBTEST.COM (short username kudu)
I20260925 02:56:15.102916 25804 webserver.cc:460] Webserver started at http://127.2.172.126:38283/ using document root <none> and password file <none>
I20260925 02:56:15.103688 25804 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:15.103781 25804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:15.104064 25804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:15.107079 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/data/instance:
uuid: "024042e977ca4966a9169b0d974c9a1b"
format_stamp: "Formatted at 2026-09-25 02:56:15 on dist-test-slave-g1xh"
server_key: "759b040f93d6638fe7145af359e1ac16"
server_key_iv: "2b2221ea776b502b55868a26ae1fe085"
server_key_version: "encryptionkey@0"
I20260925 02:56:15.107898 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/wal/instance:
uuid: "024042e977ca4966a9169b0d974c9a1b"
format_stamp: "Formatted at 2026-09-25 02:56:15 on dist-test-slave-g1xh"
server_key: "759b040f93d6638fe7145af359e1ac16"
server_key_iv: "2b2221ea776b502b55868a26ae1fe085"
server_key_version: "encryptionkey@0"
I20260925 02:56:15.113138 25804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:56:15.116432 25820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:15.117731 25804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:15.117914 25804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/wal
uuid: "024042e977ca4966a9169b0d974c9a1b"
format_stamp: "Formatted at 2026-09-25 02:56:15 on dist-test-slave-g1xh"
server_key: "759b040f93d6638fe7145af359e1ac16"
server_key_iv: "2b2221ea776b502b55868a26ae1fe085"
server_key_version: "encryptionkey@0"
I20260925 02:56:15.118144 25804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:15.132537 25804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:15.136966 25804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:15.137307 25804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:15.156649 25804 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.126:42873
I20260925 02:56:15.156669 25872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.126:42873 every 8 connection(s)
I20260925 02:56:15.158008 25804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:15.159500 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 25804
I20260925 02:56:15.159720 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-0/wal/instance
I20260925 02:56:15.160310 2737 external_mini_cluster.cc:1428] Setting key 5fb12e25b9fc49a5cd3e70d973cb863c
I20260925 02:56:15.162755 25873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:15.172761 25873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:56:15 dist-test-slave-g1xh krb5kdc[25791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304975, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.172.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.125@KRBTEST.COM" created.
I20260925 02:56:15.199018 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.176331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42873 (local address 127.0.0.1:32810)
0925 02:56:15.177182 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.177248 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:15.180077 (+ 2829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:15.180977 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:15.180987 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:15.181789 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:15.182284 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:15.182314 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.183952 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.183960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:15.185912 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:15.185931 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.186936 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.186945 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:15.187227 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:15.187968 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:15.188007 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:15.190908 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:15.194990 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:15.195001 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:15.195011 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:15.195409 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:15.195928 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:15.195931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:15.195934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:15.196097 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:15.197133 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:15.197138 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:15.197326 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 02:56:15.197552 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 02:56:15.197943 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":330,"threads_started":1}
I20260925 02:56:15.199056 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.177384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.2.172.126:42873)
0925 02:56:15.178017 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:15.178023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:15.178063 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:56:15.180472 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:15.180478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:15.180723 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:15.180876 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:15.182525 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.183792 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.186088 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.186678 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.186827 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:15.191144 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:15.191152 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:15.191164 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:15.191200 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:15.194819 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:15.195581 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:15.195594 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:15.195600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:15.195807 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:15.196213 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:15.196220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:15.196226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:15.196982 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:15.197195 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:15.197721 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 02:56:15.198149 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":465,"thread_start_us":319,"threads_started":1}
I20260925 02:56:15.202241 25873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:56:15.203225 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.202837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:49904)
0925 02:56:15.203004 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.203118 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 02:56:15.203462 25822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
I20260925 02:56:15.205104 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.204753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:49906)
0925 02:56:15.204905 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.204980 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 02:56:15.205282 25873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:39403: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.125.keytab.
Entry for principal kudu/127.2.172.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.125.keytab.
I20260925 02:56:15.230999 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.125:39403
--webserver_interface=127.2.172.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.125.keytab
--principal=kudu/127.2.172.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:42873,127.2.172.125:39403,127.2.172.124:46825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:56:15.254664 25873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } attempt: 1
I20260925 02:56:15.256142 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.255437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:49914)
0925 02:56:15.255768 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.255915 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260925 02:56:15.257331 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.256861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:49926)
0925 02:56:15.257034 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.257140 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:56:15.257591 25873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:39403: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
I20260925 02:56:15.325873 25873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } attempt: 2
I20260925 02:56:15.327133 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.326552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:49936)
0925 02:56:15.326834 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.326945 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 02:56:15.328310 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.327886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:49950)
0925 02:56:15.328048 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.328149 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:56:15.328512 25873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:39403: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
I20260925 02:56:15.419786 25873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } attempt: 3
I20260925 02:56:15.421061 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.420514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:49956)
0925 02:56:15.420740 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.420858 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 02:56:15.422211 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.421811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:49964)
0925 02:56:15.421934 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.422001 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 02:56:15.422468 25873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:39403: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
I20260925 02:56:15.565716 25873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } attempt: 4
I20260925 02:56:15.566915 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.566413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:49972)
0925 02:56:15.566699 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.566783 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260925 02:56:15.567987 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.567642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:49984)
0925 02:56:15.567781 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.567872 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 02:56:15.568176 25873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.125:39403: Network error: Client connection negotiation failed: client connection to 127.2.172.125:39403: connect: Connection refused (error 111)
W20260925 02:56:15.571980 25880 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:56:15.572325 25880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:15.572451 25880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:15.572590 25880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:15.580014 25880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:15.580137 25880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:15.580255 25880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:15.580339 25880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:15.580447 25880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:15.590430 25880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:42873,127.2.172.125:39403,127.2.172.124:46825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.125.keytab
--principal=kudu/127.2.172.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.125:39403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.172.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:15.591825 25880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:15.593166 25880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:15.604854 25886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:15.604935 25885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:15.604911 25888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:15.606333 25880 server_base.cc:1034] running on GCE node
I20260925 02:56:15.607074 25880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:15.608760 25880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:15.610045 25880 hybrid_clock.cc:648] HybridClock initialized: now 1790304975609988 us; error 39 us; skew 500 ppm
I20260925 02:56:15.610380 25880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:15 dist-test-slave-g1xh krb5kdc[25791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304975, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:15.613633 25880 init.cc:377] Logged in from keytab as kudu/127.2.172.125@KRBTEST.COM (short username kudu)
I20260925 02:56:15.615108 25880 webserver.cc:460] Webserver started at http://127.2.172.125:46699/ using document root <none> and password file <none>
I20260925 02:56:15.615965 25880 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:15.616083 25880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:15.616405 25880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:15.619415 25880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/data/instance:
uuid: "7dbd064ba125400ea027a1d0576b0212"
format_stamp: "Formatted at 2026-09-25 02:56:15 on dist-test-slave-g1xh"
server_key: "168984130674d1e03374e79bac532577"
server_key_iv: "50404db6c7d167dd457686269927eef0"
server_key_version: "encryptionkey@0"
I20260925 02:56:15.620280 25880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/wal/instance:
uuid: "7dbd064ba125400ea027a1d0576b0212"
format_stamp: "Formatted at 2026-09-25 02:56:15 on dist-test-slave-g1xh"
server_key: "168984130674d1e03374e79bac532577"
server_key_iv: "50404db6c7d167dd457686269927eef0"
server_key_version: "encryptionkey@0"
I20260925 02:56:15.625452 25880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:56:15.628755 25895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:15.630187 25880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:15.630373 25880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/wal
uuid: "7dbd064ba125400ea027a1d0576b0212"
format_stamp: "Formatted at 2026-09-25 02:56:15 on dist-test-slave-g1xh"
server_key: "168984130674d1e03374e79bac532577"
server_key_iv: "50404db6c7d167dd457686269927eef0"
server_key_version: "encryptionkey@0"
I20260925 02:56:15.630602 25880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:15.656639 25880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:15.660934 25880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:15.661228 25880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:15.680147 25880 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.125:39403
I20260925 02:56:15.680171 25946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.125:39403 every 8 connection(s)
I20260925 02:56:15.681558 25880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/data/info.pb
I20260925 02:56:15.685439 25947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:15.689953 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 25880
I20260925 02:56:15.690171 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-1/wal/instance
I20260925 02:56:15.690721 2737 external_mini_cluster.cc:1428] Setting key 3ca3ae392c5efbca195ecdb186790f5d
I20260925 02:56:15.693379 25947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:56:15 dist-test-slave-g1xh krb5kdc[25791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304975, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.125@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:56:15.716554 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.696889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.2.172.126:42873)
0925 02:56:15.697146 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:15.697155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:15.698372 (+ 1217us) server_negotiation.cc:407] Connection header received
0925 02:56:15.700377 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:15.700386 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:15.700486 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:15.700610 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:15.702100 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.703061 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.705545 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.705982 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.706083 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:15.710441 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:15.710446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:15.710450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:15.710485 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:15.713196 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:15.713994 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:15.714006 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:15.714008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:15.714135 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:15.714592 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:15.714597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:15.714601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:15.714895 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:15.715056 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:15.715970 (+ 914us) server_negotiation.cc:299] Negotiation successful
0925 02:56:15.716215 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:56:15.717545 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.697374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42873 (local address 127.0.0.1:32818)
0925 02:56:15.698174 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.698221 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:15.700094 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:15.700800 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:15.700807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:15.701358 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:15.701912 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:15.701928 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.703236 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.703244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:15.705386 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:15.705405 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.706233 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.706246 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:15.706598 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:15.707357 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:15.707398 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:15.710182 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:15.713361 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:15.713371 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:15.713391 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:15.713855 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:15.714252 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:15.714257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:15.714261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:15.714460 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:15.715029 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:15.715037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:15.715503 (+ 466us) client_negotiation.cc:769] Sending connection context
0925 02:56:15.715866 (+ 363us) client_negotiation.cc:240] Negotiation successful
0925 02:56:15.716397 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":347,"threads_started":1}
WARNING: no policy specified for kudu/127.2.172.124@KRBTEST.COM; defaulting to no policy
I20260925 02:56:15.719309 25947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.172.124@KRBTEST.COM" created.
Sep 25 02:56:15 dist-test-slave-g1xh krb5kdc[25791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304975, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.125@KRBTEST.COM for kudu/127.2.172.125@KRBTEST.COM
I20260925 02:56:15.736191 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.720393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.2.172.125:39403)
0925 02:56:15.720982 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:15.720988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:15.721013 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:56:15.721056 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:15.721062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:15.721233 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:15.721394 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:15.722298 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.724062 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.725182 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.726095 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.726239 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:15.729241 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:15.729258 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:15.729278 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:15.729325 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:15.733084 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:15.733943 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:15.733951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:15.733954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:15.734131 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:15.734575 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:15.734578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:15.734580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:15.734918 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:15.735128 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:15.735535 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 02:56:15.735881 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":453,"thread_start_us":319,"threads_started":1}
I20260925 02:56:15.736277 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.719994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:49992)
0925 02:56:15.720212 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.720241 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:15.720381 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:15.721400 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:15.721404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:15.721912 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:15.722112 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:15.722127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.724208 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.724212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:15.725027 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:15.725052 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.726235 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.726239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:15.726395 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:15.726989 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:15.727006 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:15.729014 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:15.733220 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:15.733227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:15.733232 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:15.733787 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:15.734260 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:15.734268 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:15.734275 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:15.734456 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:15.735072 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:15.735080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:15.735255 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 02:56:15.735413 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 02:56:15.735769 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":3}
I20260925 02:56:15.738637 25947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:56:15.739980 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.739349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55456)
0925 02:56:15.739637 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.739769 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 02:56:15.740288 25899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
I20260925 02:56:15.741850 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.741538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55460)
0925 02:56:15.741671 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.741749 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:56:15.742177 25947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:46825: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.172.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.124.keytab.
Entry for principal kudu/127.2.172.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.124.keytab.
I20260925 02:56:15.758744 2737 external_mini_cluster.cc:1326] Running /tmp/dist-test-task450sZB/build/asan/bin/kudu
/tmp/dist-test-task450sZB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.124:46825
--webserver_interface=127.2.172.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.124.keytab
--principal=kudu/127.2.172.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.172.84:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.172.126:42873,127.2.172.125:39403,127.2.172.124:46825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:56:15.791488 25947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } attempt: 1
I20260925 02:56:15.792961 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.792356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55476)
0925 02:56:15.792606 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.792751 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 02:56:15.794090 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.793656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55478)
0925 02:56:15.793835 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.793962 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 02:56:15.794314 25947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:46825: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
I20260925 02:56:15.862576 25947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } attempt: 2
I20260925 02:56:15.863903 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.863316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55482)
0925 02:56:15.863582 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.863704 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 02:56:15.864951 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.864550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55498)
0925 02:56:15.864713 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.864800 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 02:56:15.865185 25947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:46825: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
I20260925 02:56:15.867403 25873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } attempt: 5
Sep 25 02:56:15 dist-test-slave-g1xh krb5kdc[25791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304975, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for kudu/127.2.172.125@KRBTEST.COM
I20260925 02:56:15.881572 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.868118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:50008)
0925 02:56:15.868314 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.868336 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:15.868502 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:15.868902 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:15.868907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:15.869307 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:15.869486 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:15.869495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.870473 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.870479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:15.871351 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:15.871363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.872193 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.872199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:15.872407 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:15.873057 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:15.873075 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:15.875327 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:15.879165 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:15.879174 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:15.879179 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:15.879754 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:15.880143 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:15.880147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:15.880149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:15.880277 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:15.880843 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:15.880846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:15.880973 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 02:56:15.881095 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 02:56:15.881317 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 02:56:15.881737 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.868148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.2.172.125:39403)
0925 02:56:15.868338 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:15.868344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:15.868419 (+ 75us) server_negotiation.cc:407] Connection header received
0925 02:56:15.868682 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:15.868686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:15.868759 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:15.868866 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:15.869644 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.870336 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.871508 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:15.872019 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:15.872174 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:15.875545 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:15.875575 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:15.875580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:15.875623 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:15.878982 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:15.879869 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:15.879882 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:15.879885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:15.880031 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:15.880379 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:15.880384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:15.880387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:15.880731 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:15.880885 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:15.881232 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 02:56:15.881451 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 02:56:15.883021 25873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:56:15.884224 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.883844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55508)
0925 02:56:15.884012 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.884087 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 02:56:15.885144 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.884817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55510)
0925 02:56:15.884959 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.885047 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 02:56:15.885365 25873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:46825: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
I20260925 02:56:15.936561 25873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } attempt: 1
I20260925 02:56:15.937769 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.937301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55520)
0925 02:56:15.937500 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.937588 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 02:56:15.938818 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.938470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55534)
0925 02:56:15.938631 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.938710 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 02:56:15.939034 25873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:46825: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
I20260925 02:56:15.956398 25947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } attempt: 3
I20260925 02:56:15.957635 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.957175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55538)
0925 02:56:15.957372 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.957461 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 02:56:15.958789 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:15.958348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55554)
0925 02:56:15.958605 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:15.958674 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260925 02:56:15.958999 25947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:46825: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
I20260925 02:56:16.007252 25873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } attempt: 2
I20260925 02:56:16.008435 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.007936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55560)
0925 02:56:16.008160 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:16.008265 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 02:56:16.009426 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.009078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55568)
0925 02:56:16.009228 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:16.009315 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 02:56:16.009644 25873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:46825: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
W20260925 02:56:16.098229 25954 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:56:16.098685 25954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:16.098814 25954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:16.098964 25954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 02:56:16.102213 25947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } attempt: 4
I20260925 02:56:16.103483 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.102987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55574)
0925 02:56:16.103238 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:16.103327 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 02:56:16.104456 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.104169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55578)
0925 02:56:16.104296 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:16.104359 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 02:56:16.104658 25947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:46825: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
W20260925 02:56:16.107164 25954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:16.107286 25954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:16.107412 25954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:16.107503 25954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:16.107615 25954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:16.115841 25873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } attempt: 3
I20260925 02:56:16.116888 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.116451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55586)
0925 02:56:16.116654 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:16.116743 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 02:56:16.117694 25954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.172.84:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.172.126:42873,127.2.172.125:39403,127.2.172.124:46825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/krb5kdc/kudu_127.2.172.124.keytab
--principal=kudu/127.2.172.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.172.124:46825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.172.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:16.117856 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.117485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55602)
0925 02:56:16.117693 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:16.117757 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260925 02:56:16.118114 25873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.172.124:46825: Network error: Client connection negotiation failed: client connection to 127.2.172.124:46825: connect: Connection refused (error 111)
I20260925 02:56:16.119086 25954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:16.120414 25954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:16.136173 25959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:16.136269 25963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:16.136278 25960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:16.137036 25954 server_base.cc:1034] running on GCE node
I20260925 02:56:16.137789 25954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:16.139200 25954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:16.140415 25954 hybrid_clock.cc:648] HybridClock initialized: now 1790304976140348 us; error 41 us; skew 500 ppm
I20260925 02:56:16.140759 25954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:16 dist-test-slave-g1xh krb5kdc[25791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304976, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:16.143946 25954 init.cc:377] Logged in from keytab as kudu/127.2.172.124@KRBTEST.COM (short username kudu)
I20260925 02:56:16.145359 25954 webserver.cc:460] Webserver started at http://127.2.172.124:34829/ using document root <none> and password file <none>
I20260925 02:56:16.146054 25954 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:16.146168 25954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:16.146507 25954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:16.149514 25954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/data/instance:
uuid: "0d36940e2195408ca9e42d5f4cb80849"
format_stamp: "Formatted at 2026-09-25 02:56:16 on dist-test-slave-g1xh"
server_key: "d4e2e379f2a2488203ab84e44de79fd5"
server_key_iv: "b601a35764e576c72560185fce3e25b6"
server_key_version: "encryptionkey@0"
I20260925 02:56:16.150296 25954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/wal/instance:
uuid: "0d36940e2195408ca9e42d5f4cb80849"
format_stamp: "Formatted at 2026-09-25 02:56:16 on dist-test-slave-g1xh"
server_key: "d4e2e379f2a2488203ab84e44de79fd5"
server_key_iv: "b601a35764e576c72560185fce3e25b6"
server_key_version: "encryptionkey@0"
I20260925 02:56:16.155360 25954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:56:16.158810 25970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:16.160192 25954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:16.160364 25954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/data,/tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/wal
uuid: "0d36940e2195408ca9e42d5f4cb80849"
format_stamp: "Formatted at 2026-09-25 02:56:16 on dist-test-slave-g1xh"
server_key: "d4e2e379f2a2488203ab84e44de79fd5"
server_key_iv: "b601a35764e576c72560185fce3e25b6"
server_key_version: "encryptionkey@0"
I20260925 02:56:16.160574 25954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:16.182399 25954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:16.186537 25954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:16.186818 25954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:16.205351 25954 rpc_server.cc:307] RPC server started. Bound to: 127.2.172.124:46825
I20260925 02:56:16.205369 26022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.172.124:46825 every 8 connection(s)
I20260925 02:56:16.206707 25954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/data/info.pb
I20260925 02:56:16.210345 26023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:16.214787 2737 external_mini_cluster.cc:1388] Started /tmp/dist-test-task450sZB/build/asan/bin/kudu as pid 25954
I20260925 02:56:16.214982 2737 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task450sZB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304596269475-2737-0/minicluster-data/master-2/wal/instance
I20260925 02:56:16.215391 2737 external_mini_cluster.cc:1428] Setting key fec8c953d88862a82981aece67cdb5ff
I20260925 02:56:16.217937 26023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:56:16 dist-test-slave-g1xh krb5kdc[25791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.124@KRBTEST.COM
Sep 25 02:56:16 dist-test-slave-g1xh krb5kdc[25791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304976, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.124@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:56:16.238288 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.222005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.2.172.126:42873)
0925 02:56:16.223249 (+ 1244us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.223256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.223276 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:56:16.223379 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.223386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.223454 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.223583 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.224728 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.225510 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.226876 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.227286 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.227425 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.231239 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:16.231245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:16.231251 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:16.231292 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:16.234860 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.235898 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.235902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.235904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.236036 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.236479 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.236483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.236486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.236791 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:16.236952 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:16.237663 (+ 711us) server_negotiation.cc:299] Negotiation successful
0925 02:56:16.237879 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":1022,"threads_started":1}
I20260925 02:56:16.239261 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.221991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.126:42873 (local address 127.0.0.1:32834)
0925 02:56:16.222690 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:16.222737 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:16.223245 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:16.223662 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:16.223668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:16.224305 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:16.224578 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:16.224587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.225677 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.225682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:16.226721 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:16.226732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.227524 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.227532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:16.227711 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.228297 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:16.228318 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:16.231047 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:16.235013 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:16.235027 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:16.235041 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:16.235733 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:16.236177 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:16.236180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:16.236182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:16.236338 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:16.236914 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:16.236941 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:16.237199 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 02:56:16.237524 (+ 325us) client_negotiation.cc:240] Negotiation successful
0925 02:56:16.238189 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":256,"thread_start_us":274,"threads_started":1}
I20260925 02:56:16.239279 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.217348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.2.172.124:46825)
0925 02:56:16.218189 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.218200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.218243 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:56:16.219976 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.220001 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.220430 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.220748 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.222812 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.224365 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.225603 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.226287 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.226482 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.230818 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:16.230826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:16.230838 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:16.230872 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:16.234528 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.235440 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.235447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.235450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.235709 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.236134 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.236139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.236143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.236864 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:16.237147 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:16.237462 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 02:56:16.237894 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":377,"threads_started":1}
I20260925 02:56:16.241227 26023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:56:16 dist-test-slave-g1xh krb5kdc[25791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304976, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.124@KRBTEST.COM for kudu/127.2.172.125@KRBTEST.COM
Sep 25 02:56:16 dist-test-slave-g1xh krb5kdc[25791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.126@KRBTEST.COM
I20260925 02:56:16.256278 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.242055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.2.172.125:39403)
0925 02:56:16.242320 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.242324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.242340 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:56:16.242414 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.242419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.242510 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.242616 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.243457 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.244639 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.246313 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.246994 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.247130 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.250525 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:16.250534 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:16.250539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:16.250589 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:16.253500 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.254336 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.254341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.254343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.254482 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.254938 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.254942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.254944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.255225 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:16.255340 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:16.255831 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 02:56:16.256000 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260925 02:56:16.256412 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.241976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.125:39403 (local address 127.0.0.1:50010)
0925 02:56:16.242140 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:16.242162 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:16.242255 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:16.242645 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:16.242649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:16.243081 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:16.243282 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:16.243296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.244802 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.244806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:16.246137 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:16.246152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.247162 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.247167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:16.247303 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.247979 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:16.248005 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:16.250306 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:16.253649 (+ 3343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:16.253656 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:16.253660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:16.254201 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:16.254608 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:16.254615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:16.254621 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:16.254816 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:16.255352 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:16.255359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:16.255529 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 02:56:16.255726 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 02:56:16.256023 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 02:56:16.257550 26023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:56:16.259625 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.243225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32844 (local address 127.2.172.126:42873)
0925 02:56:16.243417 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.243422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.243440 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:56:16.243489 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.243492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.243603 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.243734 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.245113 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.246147 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.247854 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.248529 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.248664 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.253142 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:16.253147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:16.253150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:16.253184 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:16.256372 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.257261 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.257265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.257286 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.257448 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.257903 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.257906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.257908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.258302 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:16.258645 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:16.258959 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 02:56:16.259221 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 02:56:16 dist-test-slave-g1xh krb5kdc[25791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304976, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.124@KRBTEST.COM for kudu/127.2.172.124@KRBTEST.COM
Sep 25 02:56:16 dist-test-slave-g1xh krb5kdc[25791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.172.125@KRBTEST.COM
I20260925 02:56:16.272933 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.258677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55624)
0925 02:56:16.258846 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:16.258865 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:16.258955 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:16.259275 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:16.259280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:16.260157 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:16.260291 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:16.260299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.261363 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.261370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:16.262368 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:16.262379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.263066 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.263071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:16.263266 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.263893 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:16.263916 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:16.266669 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:16.270266 (+ 3597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:16.270272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:16.270276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:16.270813 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:16.271174 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:16.271182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:16.271185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:16.271316 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:16.271897 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:16.271901 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:16.272046 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 02:56:16.272210 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 02:56:16.272517 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 02:56:16.272962 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.258616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.2.172.124:46825)
0925 02:56:16.258971 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.258977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.258996 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:56:16.259042 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.259046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.259128 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.259227 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.260423 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.261220 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.262506 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.262913 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.263052 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.267182 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:16.267187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:16.267190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:16.267218 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:16.270142 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.270930 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.270933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.270935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.271055 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.271421 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.271429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.271435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.271786 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:16.271937 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:16.272307 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 02:56:16.272673 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":248}
I20260925 02:56:16.276691 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.261306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.2.172.125:39403)
0925 02:56:16.261478 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.261484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.261500 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:56:16.261564 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.261569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.261642 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.261732 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.262881 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.263730 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.265530 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.266173 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.266308 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.271225 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:16.271229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:16.271232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:16.271264 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:16.274012 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.274825 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.274829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.274832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.274962 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.275410 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.275413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.275415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.275734 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:16.275943 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:16.276269 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 02:56:16.276435 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:56:16.279143 26023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849: Bootstrap starting.
I20260925 02:56:16.281616 2737 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:16.283087 26023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:16.284448 26023 log.cc:824] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:16.287074 26023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849: No bootstrap required, opened a new log
I20260925 02:56:16.284694 25848 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "024042e977ca4966a9169b0d974c9a1b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32844
I20260925 02:56:16.294194 26023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } }
I20260925 02:56:16.294662 26023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d36940e2195408ca9e42d5f4cb80849, State: Initialized, Role: FOLLOWER
I20260925 02:56:16.295181 26023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } }
I20260925 02:56:16.295348 25873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } attempt: 4
I20260925 02:56:16.296290 26030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } } }
I20260925 02:56:16.297260 26030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:16.297490 26023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 02:56:16 dist-test-slave-g1xh krb5kdc[25791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304975, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.126@KRBTEST.COM for kudu/127.2.172.124@KRBTEST.COM
W20260925 02:56:16.308130 26041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:56:16.308269 26041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:56:16.312495 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.296058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55640)
0925 02:56:16.296252 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:16.296278 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:16.296410 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:16.296886 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:16.296890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:16.297419 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:16.297645 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:16.297657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.298747 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.298754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:16.300270 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:16.300285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.301019 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.301026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:16.301225 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.301897 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:16.301922 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:16.304814 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:16.309533 (+ 4719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:16.309540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:16.309546 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:16.310064 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:16.310512 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:16.310517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:16.310521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:16.310695 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:16.311426 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:16.311432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:16.311638 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 02:56:16.311796 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 02:56:16.312099 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 02:56:16.313479 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.296395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.2.172.124:46825)
0925 02:56:16.296574 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.296582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.296601 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:56:16.296656 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.296663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.296735 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.296827 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.297799 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.298597 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.300412 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.300863 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.301003 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.305087 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:16.305097 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:16.305104 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:16.305158 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:16.309381 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.310196 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.310202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.310205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.310368 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.310825 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.310833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.310837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.311290 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:16.311459 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:16.312015 (+ 556us) server_negotiation.cc:299] Negotiation successful
0925 02:56:16.312415 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
I20260925 02:56:16.305696 25954 master_runner.cc:186] Error getting master registration for 127.2.172.126:42873: OK, instance_id { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" instance_seqno: 1790304975151085 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:56:16.315227 25954 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:56:16.319748 25873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b: Bootstrap starting.
I20260925 02:56:16.323844 25873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:16.325169 25873 log.cc:824] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:16.328617 25873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b: No bootstrap required, opened a new log
I20260925 02:56:16.336503 25873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } }
I20260925 02:56:16.336949 25873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 024042e977ca4966a9169b0d974c9a1b, State: Initialized, Role: FOLLOWER
I20260925 02:56:16.337479 25873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } }
I20260925 02:56:16.338591 26044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } } }
I20260925 02:56:16.338984 26044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:16.339936 25873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:16.339987 25848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 02:56:16.340196 25848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:16.343187 25848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } }
I20260925 02:56:16.344725 25848 leader_election.cc:288] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [CANDIDATE]: Term 1 election: Requested vote from peers 7dbd064ba125400ea027a1d0576b0212 (127.2.172.125:39403), 0d36940e2195408ca9e42d5f4cb80849 (127.2.172.124:46825)
I20260925 02:56:16.345441 25998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "024042e977ca4966a9169b0d974c9a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0d36940e2195408ca9e42d5f4cb80849"
I20260925 02:56:16.345980 25998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:16.349538 25998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 024042e977ca4966a9169b0d974c9a1b in term 1.
I20260925 02:56:16.350212 25824 leader_election.cc:302] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 024042e977ca4966a9169b0d974c9a1b, 0d36940e2195408ca9e42d5f4cb80849; no voters:
I20260925 02:56:16.350688 26044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:16.355822 26044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [term 1 LEADER]: Becoming Leader. State: Replica: 024042e977ca4966a9169b0d974c9a1b, State: Running, Role: LEADER
I20260925 02:56:16.356421 26044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } }
I20260925 02:56:16.366465 26055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 024042e977ca4966a9169b0d974c9a1b. Latest consensus state: current_term: 1 leader_uuid: "024042e977ca4966a9169b0d974c9a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } } }
I20260925 02:56:16.366919 26055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:16.377925 26060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:16.381219 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.349425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.2.172.126:42873)
0925 02:56:16.349621 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.349627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.350228 (+ 601us) server_negotiation.cc:407] Connection header received
0925 02:56:16.350908 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.350918 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.351029 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.351125 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.355875 (+ 4750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.357269 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.361480 (+ 4211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.362402 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.362533 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.371247 (+ 8714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:16.371254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:16.371258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:16.371327 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:16.375178 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.377561 (+ 2383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.377567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.377571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.377748 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.378291 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.378296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.378299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.378718 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:16.379004 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:16.380547 (+ 1543us) server_negotiation.cc:299] Negotiation successful
0925 02:56:16.380786 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 02:56:16.390100 25826 tablet.cc:2369] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:16.397809 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.355716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.2.172.124:46825)
0925 02:56:16.355890 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.355895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.371498 (+ 15603us) server_negotiation.cc:407] Connection header received
0925 02:56:16.371808 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.371815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.371952 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.372039 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.376879 (+ 4840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.377958 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.380082 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.380778 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.380898 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.390027 (+ 9129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:16.390035 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:16.390039 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:16.390091 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:16.393752 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.394918 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.394925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.394928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.395086 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.395867 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.395874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.395879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.396293 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:16.396551 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:16.397148 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 02:56:16.397395 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:56:16.403503 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.350087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.2.172.125:39403)
0925 02:56:16.350267 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.350274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.366189 (+ 15915us) server_negotiation.cc:407] Connection header received
0925 02:56:16.366568 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.366576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.366669 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.366749 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.383122 (+ 16373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.384248 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.386056 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.386830 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.386972 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.393524 (+ 6552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:16.393532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:16.393537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:16.393590 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:16.397377 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.401092 (+ 3715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.401099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.401105 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.401287 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.401729 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.401734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.401737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.402024 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:16.402298 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:16.402671 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 02:56:16.403032 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:56:16.403870 25947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } attempt: 5
I20260925 02:56:16.407653 26060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:56:16 dist-test-slave-g1xh krb5kdc[25791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304975, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.172.125@KRBTEST.COM for kudu/127.2.172.124@KRBTEST.COM
I20260925 02:56:16.416566 25998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [term 1 FOLLOWER]: Refusing update from remote peer 024042e977ca4966a9169b0d974c9a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:16.417647 26055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:16.416446 25804 master_runner.cc:186] Error getting master registration for 127.2.172.125:39403: OK, instance_id { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" instance_seqno: 1790304975674764 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:56:16.430809 25804 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 02:56:16.438866 25976 tablet.cc:2369] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:16.440009 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.404369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.172.124:46825 (local address 127.0.0.1:55664)
0925 02:56:16.404723 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:16.404741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:16.404835 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:16.405685 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:16.405691 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:16.406213 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:16.406366 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:16.406374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.407861 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.407866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:16.409391 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:16.409411 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.410381 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.410388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:16.410674 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.412111 (+ 1437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:16.412134 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:16.431430 (+ 19296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:16.436584 (+ 5154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:16.436594 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:16.436601 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:16.437320 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:16.437910 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:16.437918 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:16.437924 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:16.438127 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:16.438904 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:16.438911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:16.439122 (+ 211us) client_negotiation.cc:769] Sending connection context
0925 02:56:16.439299 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 02:56:16.439627 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260925 02:56:16.440217 26030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 024042e977ca4966a9169b0d974c9a1b. Latest consensus state: current_term: 1 leader_uuid: "024042e977ca4966a9169b0d974c9a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } } }
I20260925 02:56:16.440553 26030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:16.441092 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:16.405042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.2.172.124:46825)
0925 02:56:16.405218 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.405223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.405240 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:56:16.405292 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.405296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.405366 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.405461 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.406525 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.407681 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.409574 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:16.410221 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:16.410365 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:16.431690 (+ 21325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:16.431700 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:16.431707 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:16.431749 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:16.436351 (+ 4602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.437504 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.437517 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.437522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.437745 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:16.438252 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:16.438257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:16.438259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:16.438645 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:16.438824 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:16.440434 (+ 1610us) server_negotiation.cc:299] Negotiation successful
0925 02:56:16.440685 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:56:16.444262 26067 mvcc.cc:205] Tried to move back new op lower bound from 7333089183392366592 to 7333089183161315328. Current Snapshot: MvccSnapshot[applied={T|T < 7333089183392366592}]
I20260925 02:56:16.445327 26055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "024042e977ca4966a9169b0d974c9a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } } }
I20260925 02:56:16.445601 26055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:16.446696 25947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212: Bootstrap starting.
I20260925 02:56:16.446969 26060 catalog_manager.cc:1347] Generated new cluster ID: 5180a1c7344b4856882f299cc410990d
I20260925 02:56:16.447072 26060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:16.451483 25947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:16.452366 26030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "024042e977ca4966a9169b0d974c9a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } } }
I20260925 02:56:16.452641 26030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:16.454279 25947 log.cc:824] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:16.457794 25947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212: No bootstrap required, opened a new log
I20260925 02:56:16.469476 25947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } }
I20260925 02:56:16.469974 25947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dbd064ba125400ea027a1d0576b0212, State: Initialized, Role: FOLLOWER
I20260925 02:56:16.470577 25947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } }
I20260925 02:56:16.472004 26073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } } }
I20260925 02:56:16.472995 26060 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:16.473153 26073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:16.474936 26060 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:16.486449 25947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:16.490227 25921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:16.494182 25921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [term 1 FOLLOWER]: Refusing update from remote peer 024042e977ca4966a9169b0d974c9a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:16.495328 26055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:16.495445 25922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "024042e977ca4966a9169b0d974c9a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7dbd064ba125400ea027a1d0576b0212"
I20260925 02:56:16.499176 25922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 024042e977ca4966a9169b0d974c9a1b in term 1.
I20260925 02:56:16.502904 26060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b: Generated new TSK 0
I20260925 02:56:16.504058 26060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:16.527678 26077 mvcc.cc:205] Tried to move back new op lower bound from 7333089183602061312 to 7333089183161315328. Current Snapshot: MvccSnapshot[applied={T|T < 7333089183602061312 or (T in {7333089183602061312})}]
I20260925 02:56:16.533668 26073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 024042e977ca4966a9169b0d974c9a1b. Latest consensus state: current_term: 1 leader_uuid: "024042e977ca4966a9169b0d974c9a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } } }
I20260925 02:56:16.533641 26080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "024042e977ca4966a9169b0d974c9a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } } }
I20260925 02:56:16.534111 26073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:16.534140 26080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:56:16.535751 26044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "024042e977ca4966a9169b0d974c9a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024042e977ca4966a9169b0d974c9a1b" member_type: VOTER last_known_addr { host: "127.2.172.126" port: 42873 } } peers { permanent_uuid: "7dbd064ba125400ea027a1d0576b0212" member_type: VOTER last_known_addr { host: "127.2.172.125" port: 39403 } } peers { permanent_uuid: "0d36940e2195408ca9e42d5f4cb80849" member_type: VOTER last_known_addr { host: "127.2.172.124" port: 46825 } } }
I20260925 02:56:16.536082 26044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024042e977ca4966a9169b0d974c9a1b [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:16.586184 26091 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260925 02:56:16.609761 26028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:16.607386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32862 (local address 127.2.172.126:42873)
0925 02:56:16.607629 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.607636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.607662 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:56:16.607786 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.607792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.607877 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.607963 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.609398 (+ 1435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32862: BlockingRecv error: recv got EOF from 127.0.0.1:32862 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 02:56:16.612746 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:16.608193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.2.172.125:39403)
0925 02:56:16.608386 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.608393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.610646 (+ 2253us) server_negotiation.cc:407] Connection header received
0925 02:56:16.610934 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.610943 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.611065 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.611149 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.612437 (+ 1288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50052: BlockingRecv error: recv got EOF from 127.0.0.1:50052 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 02:56:16.616074 26026 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:16.613739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.2.172.124:46825)
0925 02:56:16.613936 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.613942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.613962 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:56:16.614194 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.614204 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.614310 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.614403 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.615729 (+ 1326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55668: BlockingRecv error: recv got EOF from 127.0.0.1:55668 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 02:56:16.629717 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:16.623892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.2.172.125:39403)
0925 02:56:16.624118 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.624128 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.628176 (+ 4048us) server_negotiation.cc:407] Connection header received
0925 02:56:16.628439 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.628443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.628548 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.628670 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.629459 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50064: BlockingRecv error: recv got EOF from 127.0.0.1:50064 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 02:56:16.641834 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:16.633986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.2.172.125:39403)
0925 02:56:16.634236 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.634244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.640293 (+ 6049us) server_negotiation.cc:407] Connection header received
0925 02:56:16.640524 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.640529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.640619 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.640735 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.641667 (+ 932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50080: BlockingRecv error: recv got EOF from 127.0.0.1:50080 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260925 02:56:16.649756 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:16.647640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50092 (local address 127.2.172.125:39403)
0925 02:56:16.647871 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.647880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.648394 (+ 514us) server_negotiation.cc:407] Connection header received
0925 02:56:16.648617 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.648622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.648704 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.648813 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.649513 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50092: BlockingRecv error: recv got EOF from 127.0.0.1:50092 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 02:56:16.659188 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:16.656950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.2.172.125:39403)
0925 02:56:16.657187 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.657197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.657724 (+ 527us) server_negotiation.cc:407] Connection header received
0925 02:56:16.657962 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.657975 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.658070 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.658178 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.658930 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50102: BlockingRecv error: recv got EOF from 127.0.0.1:50102 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 02:56:16.672873 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:16.670692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.2.172.125:39403)
0925 02:56:16.670911 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.670920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.671425 (+ 505us) server_negotiation.cc:407] Connection header received
0925 02:56:16.671709 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.671715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.671803 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.671914 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.672639 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50104: BlockingRecv error: recv got EOF from 127.0.0.1:50104 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 02:56:16.714402 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:16.712172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.2.172.125:39403)
0925 02:56:16.712402 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.712412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.712950 (+ 538us) server_negotiation.cc:407] Connection header received
0925 02:56:16.713179 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.713185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.713285 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.713401 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.714162 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50112: BlockingRecv error: recv got EOF from 127.0.0.1:50112 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 02:56:16.752058 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:16.749723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.2.172.125:39403)
0925 02:56:16.749978 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.749988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.750576 (+ 588us) server_negotiation.cc:407] Connection header received
0925 02:56:16.750826 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.750831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.750907 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.751018 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.751779 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50124: BlockingRecv error: recv got EOF from 127.0.0.1:50124 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260925 02:56:16.822485 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:16.820076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.2.172.125:39403)
0925 02:56:16.820287 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.820295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.820852 (+ 557us) server_negotiation.cc:407] Connection header received
0925 02:56:16.821090 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.821097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.821202 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.821320 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.822262 (+ 942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50134: BlockingRecv error: recv got EOF from 127.0.0.1:50134 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 02:56:16.956539 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:16.954584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.2.172.125:39403)
0925 02:56:16.954858 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:16.954865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:16.955155 (+ 290us) server_negotiation.cc:407] Connection header received
0925 02:56:16.955303 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:16.955308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:16.955397 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:16.955489 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:16.956284 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50136: BlockingRecv error: recv got EOF from 127.0.0.1:50136 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260925 02:56:17.221261 25951 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:17.219237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.2.172.125:39403)
0925 02:56:17.219475 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:17.219485 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:17.219948 (+ 463us) server_negotiation.cc:407] Connection header received
0925 02:56:17.220137 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:17.220143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:17.220250 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:17.220366 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:17.221049 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50138: BlockingRecv error: recv got EOF from 127.0.0.1:50138 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:56:17.310346 26041 catalog_manager.cc:1259] Loaded cluster ID: 5180a1c7344b4856882f299cc410990d
I20260925 02:56:17.310461 26041 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849: loading cluster ID for follower catalog manager: success
I20260925 02:56:17.313019 26041 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849: acquiring CA information for follower catalog manager: success
I20260925 02:56:17.314715 26041 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d36940e2195408ca9e42d5f4cb80849: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:56:17.591063 26091 catalog_manager.cc:1259] Loaded cluster ID: 5180a1c7344b4856882f299cc410990d
I20260925 02:56:17.591224 26091 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212: loading cluster ID for follower catalog manager: success
I20260925 02:56:17.594211 26091 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212: acquiring CA information for follower catalog manager: success
I20260925 02:56:17.596396 26091 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7dbd064ba125400ea027a1d0576b0212: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:56:17.744849 26156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:17.737065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50148 (local address 127.2.172.125:39403)
0925 02:56:17.737730 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:17.737737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:17.737988 (+ 251us) server_negotiation.cc:407] Connection header received
0925 02:56:17.738208 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:17.738212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:17.738373 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:17.738530 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:17.739190 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:17.740146 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:17.742159 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:17.742633 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:17.742731 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:17.743396 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:17.744082 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:17.744197 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:17.744410 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 02:56:17.744518 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":315,"threads_started":1}
I20260925 02:56:17.757239 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:17.751413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.2.172.124:46825)
0925 02:56:17.752187 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:17.752195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:17.752214 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:56:17.752313 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:17.752317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:17.752460 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:17.752629 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:17.753038 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:17.753749 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:17.755075 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:17.755451 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:17.755603 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:17.756099 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:17.756651 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:17.756810 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:17.756870 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 02:56:17.756949 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":503,"threads_started":1}
I20260925 02:56:17.769614 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:17.763031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.2.172.126:42873)
0925 02:56:17.763597 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:17.763606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:17.764209 (+ 603us) server_negotiation.cc:407] Connection header received
0925 02:56:17.764565 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:17.764570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:17.764781 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:17.764975 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:17.765445 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:17.766141 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:17.767500 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:17.767959 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:17.768107 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:17.768618 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:17.768928 (+ 310us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:17.769051 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:17.769208 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 02:56:17.769290 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":238,"threads_started":1}
I20260925 02:56:17.775379 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:17.762982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.2.172.124:46825)
0925 02:56:17.763156 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:17.763161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:17.769541 (+ 6380us) server_negotiation.cc:407] Connection header received
0925 02:56:17.769798 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:17.769805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:17.769904 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:17.769953 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:17.770427 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:17.771086 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:17.772850 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:17.773431 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:17.773522 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:17.774599 (+ 1077us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:17.774727 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:17.774793 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:17.775047 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 02:56:17.775128 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:56:17.777465 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 25804
I20260925 02:56:17.778687 25868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:17.779557 26156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:17.772358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.2.172.125:39403)
0925 02:56:17.772559 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:17.772565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:17.772943 (+ 378us) server_negotiation.cc:407] Connection header received
0925 02:56:17.773225 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:17.773230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:17.773318 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:17.773379 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:17.774356 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:17.775533 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:17.777480 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:17.778055 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:17.778191 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:17.778716 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:17.778861 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:17.778961 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:17.779174 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 02:56:17.779272 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:56:17.934993 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 25804
I20260925 02:56:17.958092 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 25880
I20260925 02:56:17.959370 25942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:18.076109 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 25880
I20260925 02:56:18.095345 2737 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task450sZB/build/asan/bin/kudu pid 25954
I20260925 02:56:18.096517 26018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:18.204643 2737 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task450sZB/build/asan/bin/kudu with pid 25954
2026-09-25T02:56:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4913 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9924 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (381848 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST